Computer_Science/Visual Intelligence

39일차 - 논문 수업 (GAN)

2021. 11. 16. 19:36

728x90

Image-to-Image Translation with Conditional Adversarial Networks

Conditional GAN

GAN에서 조건을 부여하면 이미지를 생성시키는 모델
이미지를 변환할 때 생성 모델과 판별 모델에 조건으로 이미지를 넣어 학습하면
생성 모델은 입력 이미지에 따르는 결과 이미지를 생성할 수 있게 된다

흑백 이미지를 컬러와 하거나, 윤곽이 주어지면 물체를 그리는 문제와 같은 것들을 해결 할 수 있다

픽셀 하나 하나를 바꾼다

Generator

Encoder와 decoder가 skip connection(combination)으로 연결되어 있는 U-net을 사용한다
이미지의 맥락을 보존시키는데 효과적인 구조이다

Discriminator

기존의 discriminator는 이미지 전체에 대해 fake/real 판단을 내리는 반면
PatchGAN은 이미지를 작은 patch들로 쪼개 그 patch에 대해 fake/real를 판단한다
이렇게 함으로써 L1 loss가 만들어낸 전체적인 특징과 더불어 좀 더 디테일한 특징들을 이미지에서 살려낼 수 있다

부가적 이점으로는 전체 이미지보다 작은 크기에 적용하기 때문에 파라미터 수가 적고 좀 더 빠른 속도로 연산이 가능하다
또한 더 큰 사이즈의 이미지에 대해 일반화 하여 적용시킬 수 있는 장점이 있다

PatchGAN discriminator

원본 이미지에 70x70만큼 움직이면서 window연산을 하고 연산 결과 30x30 크기 이미지가 나온다
이때 30x30 크기 이미지를 통해 각각의 부분을 예측한다

convolution은 특징이 있는지 없는지 여부를 찾고 주변 데이터와의 상관관계를 따지지 않기 때문에
전체 이미지를 통해 예측을 하게 되면 정확한 예측이 불가능하다
따라서 부분 부분의 이미지를 통해 예측을 하는 방식을 사용해야 한다

import tensorflow as tf 

resnet = tf.keras.applications.ResNet152V2()

resnet.summary() # Add를 사용해서 결과를 유지했다

Model: "resnet152v2"
__________________________________________________________________________________________________
Layer (type)                    Output Shape         Param #     Connected to                     
==================================================================================================
input_2 (InputLayer)            [(None, 224, 224, 3) 0                                            
__________________________________________________________________________________________________
conv1_pad (ZeroPadding2D)       (None, 230, 230, 3)  0           input_2[0][0]                    
__________________________________________________________________________________________________
conv1_conv (Conv2D)             (None, 112, 112, 64) 9472        conv1_pad[0][0]                  
__________________________________________________________________________________________________
pool1_pad (ZeroPadding2D)       (None, 114, 114, 64) 0           conv1_conv[0][0]                 
__________________________________________________________________________________________________
pool1_pool (MaxPooling2D)       (None, 56, 56, 64)   0           pool1_pad[0][0]                  
__________________________________________________________________________________________________
conv2_block1_preact_bn (BatchNo (None, 56, 56, 64)   256         pool1_pool[0][0]                 
__________________________________________________________________________________________________
conv2_block1_preact_relu (Activ (None, 56, 56, 64)   0           conv2_block1_preact_bn[0][0]     
__________________________________________________________________________________________________
conv2_block1_1_conv (Conv2D)    (None, 56, 56, 64)   4096        conv2_block1_preact_relu[0][0]   
__________________________________________________________________________________________________
conv2_block1_1_bn (BatchNormali (None, 56, 56, 64)   256         conv2_block1_1_conv[0][0]        
__________________________________________________________________________________________________
conv2_block1_1_relu (Activation (None, 56, 56, 64)   0           conv2_block1_1_bn[0][0]          
__________________________________________________________________________________________________
conv2_block1_2_pad (ZeroPadding (None, 58, 58, 64)   0           conv2_block1_1_relu[0][0]        
__________________________________________________________________________________________________
conv2_block1_2_conv (Conv2D)    (None, 56, 56, 64)   36864       conv2_block1_2_pad[0][0]         
__________________________________________________________________________________________________
conv2_block1_2_bn (BatchNormali (None, 56, 56, 64)   256         conv2_block1_2_conv[0][0]        
__________________________________________________________________________________________________
conv2_block1_2_relu (Activation (None, 56, 56, 64)   0           conv2_block1_2_bn[0][0]          
__________________________________________________________________________________________________
conv2_block1_0_conv (Conv2D)    (None, 56, 56, 256)  16640       conv2_block1_preact_relu[0][0]   
__________________________________________________________________________________________________
conv2_block1_3_conv (Conv2D)    (None, 56, 56, 256)  16640       conv2_block1_2_relu[0][0]        
__________________________________________________________________________________________________
conv2_block1_out (Add)          (None, 56, 56, 256)  0           conv2_block1_0_conv[0][0]        
                                                                 conv2_block1_3_conv[0][0]        
__________________________________________________________________________________________________
conv2_block2_preact_bn (BatchNo (None, 56, 56, 256)  1024        conv2_block1_out[0][0]           
__________________________________________________________________________________________________
conv2_block2_preact_relu (Activ (None, 56, 56, 256)  0           conv2_block2_preact_bn[0][0]     
__________________________________________________________________________________________________
conv2_block2_1_conv (Conv2D)    (None, 56, 56, 64)   16384       conv2_block2_preact_relu[0][0]   
__________________________________________________________________________________________________
conv2_block2_1_bn (BatchNormali (None, 56, 56, 64)   256         conv2_block2_1_conv[0][0]        
__________________________________________________________________________________________________
conv2_block2_1_relu (Activation (None, 56, 56, 64)   0           conv2_block2_1_bn[0][0]          
__________________________________________________________________________________________________
conv2_block2_2_pad (ZeroPadding (None, 58, 58, 64)   0           conv2_block2_1_relu[0][0]        
__________________________________________________________________________________________________
conv2_block2_2_conv (Conv2D)    (None, 56, 56, 64)   36864       conv2_block2_2_pad[0][0]         
__________________________________________________________________________________________________
conv2_block2_2_bn (BatchNormali (None, 56, 56, 64)   256         conv2_block2_2_conv[0][0]        
__________________________________________________________________________________________________
conv2_block2_2_relu (Activation (None, 56, 56, 64)   0           conv2_block2_2_bn[0][0]          
__________________________________________________________________________________________________
conv2_block2_3_conv (Conv2D)    (None, 56, 56, 256)  16640       conv2_block2_2_relu[0][0]        
__________________________________________________________________________________________________
conv2_block2_out (Add)          (None, 56, 56, 256)  0           conv2_block1_out[0][0]           
                                                                 conv2_block2_3_conv[0][0]        
__________________________________________________________________________________________________
conv2_block3_preact_bn (BatchNo (None, 56, 56, 256)  1024        conv2_block2_out[0][0]           
__________________________________________________________________________________________________
conv2_block3_preact_relu (Activ (None, 56, 56, 256)  0           conv2_block3_preact_bn[0][0]     
__________________________________________________________________________________________________
conv2_block3_1_conv (Conv2D)    (None, 56, 56, 64)   16384       conv2_block3_preact_relu[0][0]   
__________________________________________________________________________________________________
conv2_block3_1_bn (BatchNormali (None, 56, 56, 64)   256         conv2_block3_1_conv[0][0]        
__________________________________________________________________________________________________
conv2_block3_1_relu (Activation (None, 56, 56, 64)   0           conv2_block3_1_bn[0][0]          
__________________________________________________________________________________________________
conv2_block3_2_pad (ZeroPadding (None, 58, 58, 64)   0           conv2_block3_1_relu[0][0]        
__________________________________________________________________________________________________
conv2_block3_2_conv (Conv2D)    (None, 28, 28, 64)   36864       conv2_block3_2_pad[0][0]         
__________________________________________________________________________________________________
conv2_block3_2_bn (BatchNormali (None, 28, 28, 64)   256         conv2_block3_2_conv[0][0]        
__________________________________________________________________________________________________
conv2_block3_2_relu (Activation (None, 28, 28, 64)   0           conv2_block3_2_bn[0][0]          
__________________________________________________________________________________________________
max_pooling2d_3 (MaxPooling2D)  (None, 28, 28, 256)  0           conv2_block2_out[0][0]           
__________________________________________________________________________________________________
conv2_block3_3_conv (Conv2D)    (None, 28, 28, 256)  16640       conv2_block3_2_relu[0][0]        
__________________________________________________________________________________________________
conv2_block3_out (Add)          (None, 28, 28, 256)  0           max_pooling2d_3[0][0]            
                                                                 conv2_block3_3_conv[0][0]        
__________________________________________________________________________________________________
conv3_block1_preact_bn (BatchNo (None, 28, 28, 256)  1024        conv2_block3_out[0][0]           
__________________________________________________________________________________________________
conv3_block1_preact_relu (Activ (None, 28, 28, 256)  0           conv3_block1_preact_bn[0][0]     
__________________________________________________________________________________________________
conv3_block1_1_conv (Conv2D)    (None, 28, 28, 128)  32768       conv3_block1_preact_relu[0][0]   
__________________________________________________________________________________________________
conv3_block1_1_bn (BatchNormali (None, 28, 28, 128)  512         conv3_block1_1_conv[0][0]        
__________________________________________________________________________________________________
conv3_block1_1_relu (Activation (None, 28, 28, 128)  0           conv3_block1_1_bn[0][0]          
__________________________________________________________________________________________________
conv3_block1_2_pad (ZeroPadding (None, 30, 30, 128)  0           conv3_block1_1_relu[0][0]        
__________________________________________________________________________________________________
conv3_block1_2_conv (Conv2D)    (None, 28, 28, 128)  147456      conv3_block1_2_pad[0][0]         
__________________________________________________________________________________________________
conv3_block1_2_bn (BatchNormali (None, 28, 28, 128)  512         conv3_block1_2_conv[0][0]        
__________________________________________________________________________________________________
conv3_block1_2_relu (Activation (None, 28, 28, 128)  0           conv3_block1_2_bn[0][0]          
__________________________________________________________________________________________________
conv3_block1_0_conv (Conv2D)    (None, 28, 28, 512)  131584      conv3_block1_preact_relu[0][0]   
__________________________________________________________________________________________________
conv3_block1_3_conv (Conv2D)    (None, 28, 28, 512)  66048       conv3_block1_2_relu[0][0]        
__________________________________________________________________________________________________
conv3_block1_out (Add)          (None, 28, 28, 512)  0           conv3_block1_0_conv[0][0]        
                                                                 conv3_block1_3_conv[0][0]        
__________________________________________________________________________________________________
conv3_block2_preact_bn (BatchNo (None, 28, 28, 512)  2048        conv3_block1_out[0][0]           
__________________________________________________________________________________________________
conv3_block2_preact_relu (Activ (None, 28, 28, 512)  0           conv3_block2_preact_bn[0][0]     
__________________________________________________________________________________________________
conv3_block2_1_conv (Conv2D)    (None, 28, 28, 128)  65536       conv3_block2_preact_relu[0][0]   
__________________________________________________________________________________________________
conv3_block2_1_bn (BatchNormali (None, 28, 28, 128)  512         conv3_block2_1_conv[0][0]        
__________________________________________________________________________________________________
conv3_block2_1_relu (Activation (None, 28, 28, 128)  0           conv3_block2_1_bn[0][0]          
__________________________________________________________________________________________________
conv3_block2_2_pad (ZeroPadding (None, 30, 30, 128)  0           conv3_block2_1_relu[0][0]        
__________________________________________________________________________________________________
conv3_block2_2_conv (Conv2D)    (None, 28, 28, 128)  147456      conv3_block2_2_pad[0][0]         
__________________________________________________________________________________________________
conv3_block2_2_bn (BatchNormali (None, 28, 28, 128)  512         conv3_block2_2_conv[0][0]        
__________________________________________________________________________________________________
conv3_block2_2_relu (Activation (None, 28, 28, 128)  0           conv3_block2_2_bn[0][0]          
__________________________________________________________________________________________________
conv3_block2_3_conv (Conv2D)    (None, 28, 28, 512)  66048       conv3_block2_2_relu[0][0]        
__________________________________________________________________________________________________
conv3_block2_out (Add)          (None, 28, 28, 512)  0           conv3_block1_out[0][0]           
                                                                 conv3_block2_3_conv[0][0]        
__________________________________________________________________________________________________
conv3_block3_preact_bn (BatchNo (None, 28, 28, 512)  2048        conv3_block2_out[0][0]           
__________________________________________________________________________________________________
conv3_block3_preact_relu (Activ (None, 28, 28, 512)  0           conv3_block3_preact_bn[0][0]     
__________________________________________________________________________________________________
conv3_block3_1_conv (Conv2D)    (None, 28, 28, 128)  65536       conv3_block3_preact_relu[0][0]   
__________________________________________________________________________________________________
conv3_block3_1_bn (BatchNormali (None, 28, 28, 128)  512         conv3_block3_1_conv[0][0]        
__________________________________________________________________________________________________
conv3_block3_1_relu (Activation (None, 28, 28, 128)  0           conv3_block3_1_bn[0][0]          
__________________________________________________________________________________________________
conv3_block3_2_pad (ZeroPadding (None, 30, 30, 128)  0           conv3_block3_1_relu[0][0]        
__________________________________________________________________________________________________
conv3_block3_2_conv (Conv2D)    (None, 28, 28, 128)  147456      conv3_block3_2_pad[0][0]         
__________________________________________________________________________________________________
conv3_block3_2_bn (BatchNormali (None, 28, 28, 128)  512         conv3_block3_2_conv[0][0]        
__________________________________________________________________________________________________
conv3_block3_2_relu (Activation (None, 28, 28, 128)  0           conv3_block3_2_bn[0][0]          
__________________________________________________________________________________________________
conv3_block3_3_conv (Conv2D)    (None, 28, 28, 512)  66048       conv3_block3_2_relu[0][0]        
__________________________________________________________________________________________________
conv3_block3_out (Add)          (None, 28, 28, 512)  0           conv3_block2_out[0][0]           
                                                                 conv3_block3_3_conv[0][0]        
__________________________________________________________________________________________________
conv3_block4_preact_bn (BatchNo (None, 28, 28, 512)  2048        conv3_block3_out[0][0]           
__________________________________________________________________________________________________
conv3_block4_preact_relu (Activ (None, 28, 28, 512)  0           conv3_block4_preact_bn[0][0]     
__________________________________________________________________________________________________
conv3_block4_1_conv (Conv2D)    (None, 28, 28, 128)  65536       conv3_block4_preact_relu[0][0]   
__________________________________________________________________________________________________
conv3_block4_1_bn (BatchNormali (None, 28, 28, 128)  512         conv3_block4_1_conv[0][0]        
__________________________________________________________________________________________________
conv3_block4_1_relu (Activation (None, 28, 28, 128)  0           conv3_block4_1_bn[0][0]          
__________________________________________________________________________________________________
conv3_block4_2_pad (ZeroPadding (None, 30, 30, 128)  0           conv3_block4_1_relu[0][0]        
__________________________________________________________________________________________________
conv3_block4_2_conv (Conv2D)    (None, 28, 28, 128)  147456      conv3_block4_2_pad[0][0]         
__________________________________________________________________________________________________
conv3_block4_2_bn (BatchNormali (None, 28, 28, 128)  512         conv3_block4_2_conv[0][0]        
__________________________________________________________________________________________________
conv3_block4_2_relu (Activation (None, 28, 28, 128)  0           conv3_block4_2_bn[0][0]          
__________________________________________________________________________________________________
conv3_block4_3_conv (Conv2D)    (None, 28, 28, 512)  66048       conv3_block4_2_relu[0][0]        
__________________________________________________________________________________________________
conv3_block4_out (Add)          (None, 28, 28, 512)  0           conv3_block3_out[0][0]           
                                                                 conv3_block4_3_conv[0][0]        
__________________________________________________________________________________________________
conv3_block5_preact_bn (BatchNo (None, 28, 28, 512)  2048        conv3_block4_out[0][0]           
__________________________________________________________________________________________________
conv3_block5_preact_relu (Activ (None, 28, 28, 512)  0           conv3_block5_preact_bn[0][0]     
__________________________________________________________________________________________________
conv3_block5_1_conv (Conv2D)    (None, 28, 28, 128)  65536       conv3_block5_preact_relu[0][0]   
__________________________________________________________________________________________________
conv3_block5_1_bn (BatchNormali (None, 28, 28, 128)  512         conv3_block5_1_conv[0][0]        
__________________________________________________________________________________________________
conv3_block5_1_relu (Activation (None, 28, 28, 128)  0           conv3_block5_1_bn[0][0]          
__________________________________________________________________________________________________
conv3_block5_2_pad (ZeroPadding (None, 30, 30, 128)  0           conv3_block5_1_relu[0][0]        
__________________________________________________________________________________________________
conv3_block5_2_conv (Conv2D)    (None, 28, 28, 128)  147456      conv3_block5_2_pad[0][0]         
__________________________________________________________________________________________________
conv3_block5_2_bn (BatchNormali (None, 28, 28, 128)  512         conv3_block5_2_conv[0][0]        
__________________________________________________________________________________________________
conv3_block5_2_relu (Activation (None, 28, 28, 128)  0           conv3_block5_2_bn[0][0]          
__________________________________________________________________________________________________
conv3_block5_3_conv (Conv2D)    (None, 28, 28, 512)  66048       conv3_block5_2_relu[0][0]        
__________________________________________________________________________________________________
conv3_block5_out (Add)          (None, 28, 28, 512)  0           conv3_block4_out[0][0]           
                                                                 conv3_block5_3_conv[0][0]        
__________________________________________________________________________________________________
conv3_block6_preact_bn (BatchNo (None, 28, 28, 512)  2048        conv3_block5_out[0][0]           
__________________________________________________________________________________________________
conv3_block6_preact_relu (Activ (None, 28, 28, 512)  0           conv3_block6_preact_bn[0][0]     
__________________________________________________________________________________________________
conv3_block6_1_conv (Conv2D)    (None, 28, 28, 128)  65536       conv3_block6_preact_relu[0][0]   
__________________________________________________________________________________________________
conv3_block6_1_bn (BatchNormali (None, 28, 28, 128)  512         conv3_block6_1_conv[0][0]        
__________________________________________________________________________________________________
conv3_block6_1_relu (Activation (None, 28, 28, 128)  0           conv3_block6_1_bn[0][0]          
__________________________________________________________________________________________________
conv3_block6_2_pad (ZeroPadding (None, 30, 30, 128)  0           conv3_block6_1_relu[0][0]        
__________________________________________________________________________________________________
conv3_block6_2_conv (Conv2D)    (None, 28, 28, 128)  147456      conv3_block6_2_pad[0][0]         
__________________________________________________________________________________________________
conv3_block6_2_bn (BatchNormali (None, 28, 28, 128)  512         conv3_block6_2_conv[0][0]        
__________________________________________________________________________________________________
conv3_block6_2_relu (Activation (None, 28, 28, 128)  0           conv3_block6_2_bn[0][0]          
__________________________________________________________________________________________________
conv3_block6_3_conv (Conv2D)    (None, 28, 28, 512)  66048       conv3_block6_2_relu[0][0]        
__________________________________________________________________________________________________
conv3_block6_out (Add)          (None, 28, 28, 512)  0           conv3_block5_out[0][0]           
                                                                 conv3_block6_3_conv[0][0]        
__________________________________________________________________________________________________
conv3_block7_preact_bn (BatchNo (None, 28, 28, 512)  2048        conv3_block6_out[0][0]           
__________________________________________________________________________________________________
conv3_block7_preact_relu (Activ (None, 28, 28, 512)  0           conv3_block7_preact_bn[0][0]     
__________________________________________________________________________________________________
conv3_block7_1_conv (Conv2D)    (None, 28, 28, 128)  65536       conv3_block7_preact_relu[0][0]   
__________________________________________________________________________________________________
conv3_block7_1_bn (BatchNormali (None, 28, 28, 128)  512         conv3_block7_1_conv[0][0]        
__________________________________________________________________________________________________
conv3_block7_1_relu (Activation (None, 28, 28, 128)  0           conv3_block7_1_bn[0][0]          
__________________________________________________________________________________________________
conv3_block7_2_pad (ZeroPadding (None, 30, 30, 128)  0           conv3_block7_1_relu[0][0]        
__________________________________________________________________________________________________
conv3_block7_2_conv (Conv2D)    (None, 28, 28, 128)  147456      conv3_block7_2_pad[0][0]         
__________________________________________________________________________________________________
conv3_block7_2_bn (BatchNormali (None, 28, 28, 128)  512         conv3_block7_2_conv[0][0]        
__________________________________________________________________________________________________
conv3_block7_2_relu (Activation (None, 28, 28, 128)  0           conv3_block7_2_bn[0][0]          
__________________________________________________________________________________________________
conv3_block7_3_conv (Conv2D)    (None, 28, 28, 512)  66048       conv3_block7_2_relu[0][0]        
__________________________________________________________________________________________________
conv3_block7_out (Add)          (None, 28, 28, 512)  0           conv3_block6_out[0][0]           
                                                                 conv3_block7_3_conv[0][0]        
__________________________________________________________________________________________________
conv3_block8_preact_bn (BatchNo (None, 28, 28, 512)  2048        conv3_block7_out[0][0]           
__________________________________________________________________________________________________
conv3_block8_preact_relu (Activ (None, 28, 28, 512)  0           conv3_block8_preact_bn[0][0]     
__________________________________________________________________________________________________
conv3_block8_1_conv (Conv2D)    (None, 28, 28, 128)  65536       conv3_block8_preact_relu[0][0]   
__________________________________________________________________________________________________
conv3_block8_1_bn (BatchNormali (None, 28, 28, 128)  512         conv3_block8_1_conv[0][0]        
__________________________________________________________________________________________________
conv3_block8_1_relu (Activation (None, 28, 28, 128)  0           conv3_block8_1_bn[0][0]          
__________________________________________________________________________________________________
conv3_block8_2_pad (ZeroPadding (None, 30, 30, 128)  0           conv3_block8_1_relu[0][0]        
__________________________________________________________________________________________________
conv3_block8_2_conv (Conv2D)    (None, 14, 14, 128)  147456      conv3_block8_2_pad[0][0]         
__________________________________________________________________________________________________
conv3_block8_2_bn (BatchNormali (None, 14, 14, 128)  512         conv3_block8_2_conv[0][0]        
__________________________________________________________________________________________________
conv3_block8_2_relu (Activation (None, 14, 14, 128)  0           conv3_block8_2_bn[0][0]          
__________________________________________________________________________________________________
max_pooling2d_4 (MaxPooling2D)  (None, 14, 14, 512)  0           conv3_block7_out[0][0]           
__________________________________________________________________________________________________
conv3_block8_3_conv (Conv2D)    (None, 14, 14, 512)  66048       conv3_block8_2_relu[0][0]        
__________________________________________________________________________________________________
conv3_block8_out (Add)          (None, 14, 14, 512)  0           max_pooling2d_4[0][0]            
                                                                 conv3_block8_3_conv[0][0]        
__________________________________________________________________________________________________
conv4_block1_preact_bn (BatchNo (None, 14, 14, 512)  2048        conv3_block8_out[0][0]           
__________________________________________________________________________________________________
conv4_block1_preact_relu (Activ (None, 14, 14, 512)  0           conv4_block1_preact_bn[0][0]     
__________________________________________________________________________________________________
conv4_block1_1_conv (Conv2D)    (None, 14, 14, 256)  131072      conv4_block1_preact_relu[0][0]   
__________________________________________________________________________________________________
conv4_block1_1_bn (BatchNormali (None, 14, 14, 256)  1024        conv4_block1_1_conv[0][0]        
__________________________________________________________________________________________________
conv4_block1_1_relu (Activation (None, 14, 14, 256)  0           conv4_block1_1_bn[0][0]          
__________________________________________________________________________________________________
conv4_block1_2_pad (ZeroPadding (None, 16, 16, 256)  0           conv4_block1_1_relu[0][0]        
__________________________________________________________________________________________________
conv4_block1_2_conv (Conv2D)    (None, 14, 14, 256)  589824      conv4_block1_2_pad[0][0]         
__________________________________________________________________________________________________
conv4_block1_2_bn (BatchNormali (None, 14, 14, 256)  1024        conv4_block1_2_conv[0][0]        
__________________________________________________________________________________________________
conv4_block1_2_relu (Activation (None, 14, 14, 256)  0           conv4_block1_2_bn[0][0]          
__________________________________________________________________________________________________
conv4_block1_0_conv (Conv2D)    (None, 14, 14, 1024) 525312      conv4_block1_preact_relu[0][0]   
__________________________________________________________________________________________________
conv4_block1_3_conv (Conv2D)    (None, 14, 14, 1024) 263168      conv4_block1_2_relu[0][0]        
__________________________________________________________________________________________________
conv4_block1_out (Add)          (None, 14, 14, 1024) 0           conv4_block1_0_conv[0][0]        
                                                                 conv4_block1_3_conv[0][0]        
__________________________________________________________________________________________________
conv4_block2_preact_bn (BatchNo (None, 14, 14, 1024) 4096        conv4_block1_out[0][0]           
__________________________________________________________________________________________________
conv4_block2_preact_relu (Activ (None, 14, 14, 1024) 0           conv4_block2_preact_bn[0][0]     
__________________________________________________________________________________________________
conv4_block2_1_conv (Conv2D)    (None, 14, 14, 256)  262144      conv4_block2_preact_relu[0][0]   
__________________________________________________________________________________________________
conv4_block2_1_bn (BatchNormali (None, 14, 14, 256)  1024        conv4_block2_1_conv[0][0]        
__________________________________________________________________________________________________
conv4_block2_1_relu (Activation (None, 14, 14, 256)  0           conv4_block2_1_bn[0][0]          
__________________________________________________________________________________________________
conv4_block2_2_pad (ZeroPadding (None, 16, 16, 256)  0           conv4_block2_1_relu[0][0]        
__________________________________________________________________________________________________
conv4_block2_2_conv (Conv2D)    (None, 14, 14, 256)  589824      conv4_block2_2_pad[0][0]         
__________________________________________________________________________________________________
conv4_block2_2_bn (BatchNormali (None, 14, 14, 256)  1024        conv4_block2_2_conv[0][0]        
__________________________________________________________________________________________________
conv4_block2_2_relu (Activation (None, 14, 14, 256)  0           conv4_block2_2_bn[0][0]          
__________________________________________________________________________________________________
conv4_block2_3_conv (Conv2D)    (None, 14, 14, 1024) 263168      conv4_block2_2_relu[0][0]        
__________________________________________________________________________________________________
conv4_block2_out (Add)          (None, 14, 14, 1024) 0           conv4_block1_out[0][0]           
                                                                 conv4_block2_3_conv[0][0]        
__________________________________________________________________________________________________
conv4_block3_preact_bn (BatchNo (None, 14, 14, 1024) 4096        conv4_block2_out[0][0]           
__________________________________________________________________________________________________
conv4_block3_preact_relu (Activ (None, 14, 14, 1024) 0           conv4_block3_preact_bn[0][0]     
__________________________________________________________________________________________________
conv4_block3_1_conv (Conv2D)    (None, 14, 14, 256)  262144      conv4_block3_preact_relu[0][0]   
__________________________________________________________________________________________________
conv4_block3_1_bn (BatchNormali (None, 14, 14, 256)  1024        conv4_block3_1_conv[0][0]        
__________________________________________________________________________________________________
conv4_block3_1_relu (Activation (None, 14, 14, 256)  0           conv4_block3_1_bn[0][0]          
__________________________________________________________________________________________________
conv4_block3_2_pad (ZeroPadding (None, 16, 16, 256)  0           conv4_block3_1_relu[0][0]        
__________________________________________________________________________________________________
conv4_block3_2_conv (Conv2D)    (None, 14, 14, 256)  589824      conv4_block3_2_pad[0][0]         
__________________________________________________________________________________________________
conv4_block3_2_bn (BatchNormali (None, 14, 14, 256)  1024        conv4_block3_2_conv[0][0]        
__________________________________________________________________________________________________
conv4_block3_2_relu (Activation (None, 14, 14, 256)  0           conv4_block3_2_bn[0][0]          
__________________________________________________________________________________________________
conv4_block3_3_conv (Conv2D)    (None, 14, 14, 1024) 263168      conv4_block3_2_relu[0][0]        
__________________________________________________________________________________________________
conv4_block3_out (Add)          (None, 14, 14, 1024) 0           conv4_block2_out[0][0]           
                                                                 conv4_block3_3_conv[0][0]        
__________________________________________________________________________________________________
conv4_block4_preact_bn (BatchNo (None, 14, 14, 1024) 4096        conv4_block3_out[0][0]           
__________________________________________________________________________________________________
conv4_block4_preact_relu (Activ (None, 14, 14, 1024) 0           conv4_block4_preact_bn[0][0]     
__________________________________________________________________________________________________
conv4_block4_1_conv (Conv2D)    (None, 14, 14, 256)  262144      conv4_block4_preact_relu[0][0]   
__________________________________________________________________________________________________
conv4_block4_1_bn (BatchNormali (None, 14, 14, 256)  1024        conv4_block4_1_conv[0][0]        
__________________________________________________________________________________________________
conv4_block4_1_relu (Activation (None, 14, 14, 256)  0           conv4_block4_1_bn[0][0]          
__________________________________________________________________________________________________
conv4_block4_2_pad (ZeroPadding (None, 16, 16, 256)  0           conv4_block4_1_relu[0][0]        
__________________________________________________________________________________________________
conv4_block4_2_conv (Conv2D)    (None, 14, 14, 256)  589824      conv4_block4_2_pad[0][0]         
__________________________________________________________________________________________________
conv4_block4_2_bn (BatchNormali (None, 14, 14, 256)  1024        conv4_block4_2_conv[0][0]        
__________________________________________________________________________________________________
conv4_block4_2_relu (Activation (None, 14, 14, 256)  0           conv4_block4_2_bn[0][0]          
__________________________________________________________________________________________________
conv4_block4_3_conv (Conv2D)    (None, 14, 14, 1024) 263168      conv4_block4_2_relu[0][0]        
__________________________________________________________________________________________________
conv4_block4_out (Add)          (None, 14, 14, 1024) 0           conv4_block3_out[0][0]           
                                                                 conv4_block4_3_conv[0][0]        
__________________________________________________________________________________________________
conv4_block5_preact_bn (BatchNo (None, 14, 14, 1024) 4096        conv4_block4_out[0][0]           
__________________________________________________________________________________________________
conv4_block5_preact_relu (Activ (None, 14, 14, 1024) 0           conv4_block5_preact_bn[0][0]     
__________________________________________________________________________________________________
conv4_block5_1_conv (Conv2D)    (None, 14, 14, 256)  262144      conv4_block5_preact_relu[0][0]   
__________________________________________________________________________________________________
conv4_block5_1_bn (BatchNormali (None, 14, 14, 256)  1024        conv4_block5_1_conv[0][0]        
__________________________________________________________________________________________________
conv4_block5_1_relu (Activation (None, 14, 14, 256)  0           conv4_block5_1_bn[0][0]          
__________________________________________________________________________________________________
conv4_block5_2_pad (ZeroPadding (None, 16, 16, 256)  0           conv4_block5_1_relu[0][0]        
__________________________________________________________________________________________________
conv4_block5_2_conv (Conv2D)    (None, 14, 14, 256)  589824      conv4_block5_2_pad[0][0]         
__________________________________________________________________________________________________
conv4_block5_2_bn (BatchNormali (None, 14, 14, 256)  1024        conv4_block5_2_conv[0][0]        
__________________________________________________________________________________________________
conv4_block5_2_relu (Activation (None, 14, 14, 256)  0           conv4_block5_2_bn[0][0]          
__________________________________________________________________________________________________
conv4_block5_3_conv (Conv2D)    (None, 14, 14, 1024) 263168      conv4_block5_2_relu[0][0]        
__________________________________________________________________________________________________
conv4_block5_out (Add)          (None, 14, 14, 1024) 0           conv4_block4_out[0][0]           
                                                                 conv4_block5_3_conv[0][0]        
__________________________________________________________________________________________________
conv4_block6_preact_bn (BatchNo (None, 14, 14, 1024) 4096        conv4_block5_out[0][0]           
__________________________________________________________________________________________________
conv4_block6_preact_relu (Activ (None, 14, 14, 1024) 0           conv4_block6_preact_bn[0][0]     
__________________________________________________________________________________________________
conv4_block6_1_conv (Conv2D)    (None, 14, 14, 256)  262144      conv4_block6_preact_relu[0][0]   
__________________________________________________________________________________________________
conv4_block6_1_bn (BatchNormali (None, 14, 14, 256)  1024        conv4_block6_1_conv[0][0]        
__________________________________________________________________________________________________
conv4_block6_1_relu (Activation (None, 14, 14, 256)  0           conv4_block6_1_bn[0][0]          
__________________________________________________________________________________________________
conv4_block6_2_pad (ZeroPadding (None, 16, 16, 256)  0           conv4_block6_1_relu[0][0]        
__________________________________________________________________________________________________
conv4_block6_2_conv (Conv2D)    (None, 14, 14, 256)  589824      conv4_block6_2_pad[0][0]         
__________________________________________________________________________________________________
conv4_block6_2_bn (BatchNormali (None, 14, 14, 256)  1024        conv4_block6_2_conv[0][0]        
__________________________________________________________________________________________________
conv4_block6_2_relu (Activation (None, 14, 14, 256)  0           conv4_block6_2_bn[0][0]          
__________________________________________________________________________________________________
conv4_block6_3_conv (Conv2D)    (None, 14, 14, 1024) 263168      conv4_block6_2_relu[0][0]        
__________________________________________________________________________________________________
conv4_block6_out (Add)          (None, 14, 14, 1024) 0           conv4_block5_out[0][0]           
                                                                 conv4_block6_3_conv[0][0]        
__________________________________________________________________________________________________
conv4_block7_preact_bn (BatchNo (None, 14, 14, 1024) 4096        conv4_block6_out[0][0]           
__________________________________________________________________________________________________
conv4_block7_preact_relu (Activ (None, 14, 14, 1024) 0           conv4_block7_preact_bn[0][0]     
__________________________________________________________________________________________________
conv4_block7_1_conv (Conv2D)    (None, 14, 14, 256)  262144      conv4_block7_preact_relu[0][0]   
__________________________________________________________________________________________________
conv4_block7_1_bn (BatchNormali (None, 14, 14, 256)  1024        conv4_block7_1_conv[0][0]        
__________________________________________________________________________________________________
conv4_block7_1_relu (Activation (None, 14, 14, 256)  0           conv4_block7_1_bn[0][0]          
__________________________________________________________________________________________________
conv4_block7_2_pad (ZeroPadding (None, 16, 16, 256)  0           conv4_block7_1_relu[0][0]        
__________________________________________________________________________________________________
conv4_block7_2_conv (Conv2D)    (None, 14, 14, 256)  589824      conv4_block7_2_pad[0][0]         
__________________________________________________________________________________________________
conv4_block7_2_bn (BatchNormali (None, 14, 14, 256)  1024        conv4_block7_2_conv[0][0]        
__________________________________________________________________________________________________
conv4_block7_2_relu (Activation (None, 14, 14, 256)  0           conv4_block7_2_bn[0][0]          
__________________________________________________________________________________________________
conv4_block7_3_conv (Conv2D)    (None, 14, 14, 1024) 263168      conv4_block7_2_relu[0][0]        
__________________________________________________________________________________________________
conv4_block7_out (Add)          (None, 14, 14, 1024) 0           conv4_block6_out[0][0]           
                                                                 conv4_block7_3_conv[0][0]        
__________________________________________________________________________________________________
conv4_block8_preact_bn (BatchNo (None, 14, 14, 1024) 4096        conv4_block7_out[0][0]           
__________________________________________________________________________________________________
conv4_block8_preact_relu (Activ (None, 14, 14, 1024) 0           conv4_block8_preact_bn[0][0]     
__________________________________________________________________________________________________
conv4_block8_1_conv (Conv2D)    (None, 14, 14, 256)  262144      conv4_block8_preact_relu[0][0]   
__________________________________________________________________________________________________
conv4_block8_1_bn (BatchNormali (None, 14, 14, 256)  1024        conv4_block8_1_conv[0][0]        
__________________________________________________________________________________________________
conv4_block8_1_relu (Activation (None, 14, 14, 256)  0           conv4_block8_1_bn[0][0]          
__________________________________________________________________________________________________
conv4_block8_2_pad (ZeroPadding (None, 16, 16, 256)  0           conv4_block8_1_relu[0][0]        
__________________________________________________________________________________________________
conv4_block8_2_conv (Conv2D)    (None, 14, 14, 256)  589824      conv4_block8_2_pad[0][0]         
__________________________________________________________________________________________________
conv4_block8_2_bn (BatchNormali (None, 14, 14, 256)  1024        conv4_block8_2_conv[0][0]        
__________________________________________________________________________________________________
conv4_block8_2_relu (Activation (None, 14, 14, 256)  0           conv4_block8_2_bn[0][0]          
__________________________________________________________________________________________________
conv4_block8_3_conv (Conv2D)    (None, 14, 14, 1024) 263168      conv4_block8_2_relu[0][0]        
__________________________________________________________________________________________________
conv4_block8_out (Add)          (None, 14, 14, 1024) 0           conv4_block7_out[0][0]           
                                                                 conv4_block8_3_conv[0][0]        
__________________________________________________________________________________________________
conv4_block9_preact_bn (BatchNo (None, 14, 14, 1024) 4096        conv4_block8_out[0][0]           
__________________________________________________________________________________________________
conv4_block9_preact_relu (Activ (None, 14, 14, 1024) 0           conv4_block9_preact_bn[0][0]     
__________________________________________________________________________________________________
conv4_block9_1_conv (Conv2D)    (None, 14, 14, 256)  262144      conv4_block9_preact_relu[0][0]   
__________________________________________________________________________________________________
conv4_block9_1_bn (BatchNormali (None, 14, 14, 256)  1024        conv4_block9_1_conv[0][0]        
__________________________________________________________________________________________________
conv4_block9_1_relu (Activation (None, 14, 14, 256)  0           conv4_block9_1_bn[0][0]          
__________________________________________________________________________________________________
conv4_block9_2_pad (ZeroPadding (None, 16, 16, 256)  0           conv4_block9_1_relu[0][0]        
__________________________________________________________________________________________________
conv4_block9_2_conv (Conv2D)    (None, 14, 14, 256)  589824      conv4_block9_2_pad[0][0]         
__________________________________________________________________________________________________
conv4_block9_2_bn (BatchNormali (None, 14, 14, 256)  1024        conv4_block9_2_conv[0][0]        
__________________________________________________________________________________________________
conv4_block9_2_relu (Activation (None, 14, 14, 256)  0           conv4_block9_2_bn[0][0]          
__________________________________________________________________________________________________
conv4_block9_3_conv (Conv2D)    (None, 14, 14, 1024) 263168      conv4_block9_2_relu[0][0]        
__________________________________________________________________________________________________
conv4_block9_out (Add)          (None, 14, 14, 1024) 0           conv4_block8_out[0][0]           
                                                                 conv4_block9_3_conv[0][0]        
__________________________________________________________________________________________________
conv4_block10_preact_bn (BatchN (None, 14, 14, 1024) 4096        conv4_block9_out[0][0]           
__________________________________________________________________________________________________
conv4_block10_preact_relu (Acti (None, 14, 14, 1024) 0           conv4_block10_preact_bn[0][0]    
__________________________________________________________________________________________________
conv4_block10_1_conv (Conv2D)   (None, 14, 14, 256)  262144      conv4_block10_preact_relu[0][0]  
__________________________________________________________________________________________________
conv4_block10_1_bn (BatchNormal (None, 14, 14, 256)  1024        conv4_block10_1_conv[0][0]       
__________________________________________________________________________________________________
conv4_block10_1_relu (Activatio (None, 14, 14, 256)  0           conv4_block10_1_bn[0][0]         
__________________________________________________________________________________________________
conv4_block10_2_pad (ZeroPaddin (None, 16, 16, 256)  0           conv4_block10_1_relu[0][0]       
__________________________________________________________________________________________________
conv4_block10_2_conv (Conv2D)   (None, 14, 14, 256)  589824      conv4_block10_2_pad[0][0]        
__________________________________________________________________________________________________
conv4_block10_2_bn (BatchNormal (None, 14, 14, 256)  1024        conv4_block10_2_conv[0][0]       
__________________________________________________________________________________________________
conv4_block10_2_relu (Activatio (None, 14, 14, 256)  0           conv4_block10_2_bn[0][0]         
__________________________________________________________________________________________________
conv4_block10_3_conv (Conv2D)   (None, 14, 14, 1024) 263168      conv4_block10_2_relu[0][0]       
__________________________________________________________________________________________________
conv4_block10_out (Add)         (None, 14, 14, 1024) 0           conv4_block9_out[0][0]           
                                                                 conv4_block10_3_conv[0][0]       
__________________________________________________________________________________________________
conv4_block11_preact_bn (BatchN (None, 14, 14, 1024) 4096        conv4_block10_out[0][0]          
__________________________________________________________________________________________________
conv4_block11_preact_relu (Acti (None, 14, 14, 1024) 0           conv4_block11_preact_bn[0][0]    
__________________________________________________________________________________________________
conv4_block11_1_conv (Conv2D)   (None, 14, 14, 256)  262144      conv4_block11_preact_relu[0][0]  
__________________________________________________________________________________________________
conv4_block11_1_bn (BatchNormal (None, 14, 14, 256)  1024        conv4_block11_1_conv[0][0]       
__________________________________________________________________________________________________
conv4_block11_1_relu (Activatio (None, 14, 14, 256)  0           conv4_block11_1_bn[0][0]         
__________________________________________________________________________________________________
conv4_block11_2_pad (ZeroPaddin (None, 16, 16, 256)  0           conv4_block11_1_relu[0][0]       
__________________________________________________________________________________________________
conv4_block11_2_conv (Conv2D)   (None, 14, 14, 256)  589824      conv4_block11_2_pad[0][0]        
__________________________________________________________________________________________________
conv4_block11_2_bn (BatchNormal (None, 14, 14, 256)  1024        conv4_block11_2_conv[0][0]       
__________________________________________________________________________________________________
conv4_block11_2_relu (Activatio (None, 14, 14, 256)  0           conv4_block11_2_bn[0][0]         
__________________________________________________________________________________________________
conv4_block11_3_conv (Conv2D)   (None, 14, 14, 1024) 263168      conv4_block11_2_relu[0][0]       
__________________________________________________________________________________________________
conv4_block11_out (Add)         (None, 14, 14, 1024) 0           conv4_block10_out[0][0]          
                                                                 conv4_block11_3_conv[0][0]       
__________________________________________________________________________________________________
conv4_block12_preact_bn (BatchN (None, 14, 14, 1024) 4096        conv4_block11_out[0][0]          
__________________________________________________________________________________________________
conv4_block12_preact_relu (Acti (None, 14, 14, 1024) 0           conv4_block12_preact_bn[0][0]    
__________________________________________________________________________________________________
conv4_block12_1_conv (Conv2D)   (None, 14, 14, 256)  262144      conv4_block12_preact_relu[0][0]  
__________________________________________________________________________________________________
conv4_block12_1_bn (BatchNormal (None, 14, 14, 256)  1024        conv4_block12_1_conv[0][0]       
__________________________________________________________________________________________________
conv4_block12_1_relu (Activatio (None, 14, 14, 256)  0           conv4_block12_1_bn[0][0]         
__________________________________________________________________________________________________
conv4_block12_2_pad (ZeroPaddin (None, 16, 16, 256)  0           conv4_block12_1_relu[0][0]       
__________________________________________________________________________________________________
conv4_block12_2_conv (Conv2D)   (None, 14, 14, 256)  589824      conv4_block12_2_pad[0][0]        
__________________________________________________________________________________________________
conv4_block12_2_bn (BatchNormal (None, 14, 14, 256)  1024        conv4_block12_2_conv[0][0]       
__________________________________________________________________________________________________
conv4_block12_2_relu (Activatio (None, 14, 14, 256)  0           conv4_block12_2_bn[0][0]         
__________________________________________________________________________________________________
conv4_block12_3_conv (Conv2D)   (None, 14, 14, 1024) 263168      conv4_block12_2_relu[0][0]       
__________________________________________________________________________________________________
conv4_block12_out (Add)         (None, 14, 14, 1024) 0           conv4_block11_out[0][0]          
                                                                 conv4_block12_3_conv[0][0]       
__________________________________________________________________________________________________
conv4_block13_preact_bn (BatchN (None, 14, 14, 1024) 4096        conv4_block12_out[0][0]          
__________________________________________________________________________________________________
conv4_block13_preact_relu (Acti (None, 14, 14, 1024) 0           conv4_block13_preact_bn[0][0]    
__________________________________________________________________________________________________
conv4_block13_1_conv (Conv2D)   (None, 14, 14, 256)  262144      conv4_block13_preact_relu[0][0]  
__________________________________________________________________________________________________
conv4_block13_1_bn (BatchNormal (None, 14, 14, 256)  1024        conv4_block13_1_conv[0][0]       
__________________________________________________________________________________________________
conv4_block13_1_relu (Activatio (None, 14, 14, 256)  0           conv4_block13_1_bn[0][0]         
__________________________________________________________________________________________________
conv4_block13_2_pad (ZeroPaddin (None, 16, 16, 256)  0           conv4_block13_1_relu[0][0]       
__________________________________________________________________________________________________
conv4_block13_2_conv (Conv2D)   (None, 14, 14, 256)  589824      conv4_block13_2_pad[0][0]        
__________________________________________________________________________________________________
conv4_block13_2_bn (BatchNormal (None, 14, 14, 256)  1024        conv4_block13_2_conv[0][0]       
__________________________________________________________________________________________________
conv4_block13_2_relu (Activatio (None, 14, 14, 256)  0           conv4_block13_2_bn[0][0]         
__________________________________________________________________________________________________
conv4_block13_3_conv (Conv2D)   (None, 14, 14, 1024) 263168      conv4_block13_2_relu[0][0]       
__________________________________________________________________________________________________
conv4_block13_out (Add)         (None, 14, 14, 1024) 0           conv4_block12_out[0][0]          
                                                                 conv4_block13_3_conv[0][0]       
__________________________________________________________________________________________________
conv4_block14_preact_bn (BatchN (None, 14, 14, 1024) 4096        conv4_block13_out[0][0]          
__________________________________________________________________________________________________
conv4_block14_preact_relu (Acti (None, 14, 14, 1024) 0           conv4_block14_preact_bn[0][0]    
__________________________________________________________________________________________________
conv4_block14_1_conv (Conv2D)   (None, 14, 14, 256)  262144      conv4_block14_preact_relu[0][0]  
__________________________________________________________________________________________________
conv4_block14_1_bn (BatchNormal (None, 14, 14, 256)  1024        conv4_block14_1_conv[0][0]       
__________________________________________________________________________________________________
conv4_block14_1_relu (Activatio (None, 14, 14, 256)  0           conv4_block14_1_bn[0][0]         
__________________________________________________________________________________________________
conv4_block14_2_pad (ZeroPaddin (None, 16, 16, 256)  0           conv4_block14_1_relu[0][0]       
__________________________________________________________________________________________________
conv4_block14_2_conv (Conv2D)   (None, 14, 14, 256)  589824      conv4_block14_2_pad[0][0]        
__________________________________________________________________________________________________
conv4_block14_2_bn (BatchNormal (None, 14, 14, 256)  1024        conv4_block14_2_conv[0][0]       
__________________________________________________________________________________________________
conv4_block14_2_relu (Activatio (None, 14, 14, 256)  0           conv4_block14_2_bn[0][0]         
__________________________________________________________________________________________________
conv4_block14_3_conv (Conv2D)   (None, 14, 14, 1024) 263168      conv4_block14_2_relu[0][0]       
__________________________________________________________________________________________________
conv4_block14_out (Add)         (None, 14, 14, 1024) 0           conv4_block13_out[0][0]          
                                                                 conv4_block14_3_conv[0][0]       
__________________________________________________________________________________________________
conv4_block15_preact_bn (BatchN (None, 14, 14, 1024) 4096        conv4_block14_out[0][0]          
__________________________________________________________________________________________________
conv4_block15_preact_relu (Acti (None, 14, 14, 1024) 0           conv4_block15_preact_bn[0][0]    
__________________________________________________________________________________________________
conv4_block15_1_conv (Conv2D)   (None, 14, 14, 256)  262144      conv4_block15_preact_relu[0][0]  
__________________________________________________________________________________________________
conv4_block15_1_bn (BatchNormal (None, 14, 14, 256)  1024        conv4_block15_1_conv[0][0]       
__________________________________________________________________________________________________
conv4_block15_1_relu (Activatio (None, 14, 14, 256)  0           conv4_block15_1_bn[0][0]         
__________________________________________________________________________________________________
conv4_block15_2_pad (ZeroPaddin (None, 16, 16, 256)  0           conv4_block15_1_relu[0][0]       
__________________________________________________________________________________________________
conv4_block15_2_conv (Conv2D)   (None, 14, 14, 256)  589824      conv4_block15_2_pad[0][0]        
__________________________________________________________________________________________________
conv4_block15_2_bn (BatchNormal (None, 14, 14, 256)  1024        conv4_block15_2_conv[0][0]       
__________________________________________________________________________________________________
conv4_block15_2_relu (Activatio (None, 14, 14, 256)  0           conv4_block15_2_bn[0][0]         
__________________________________________________________________________________________________
conv4_block15_3_conv (Conv2D)   (None, 14, 14, 1024) 263168      conv4_block15_2_relu[0][0]       
__________________________________________________________________________________________________
conv4_block15_out (Add)         (None, 14, 14, 1024) 0           conv4_block14_out[0][0]          
                                                                 conv4_block15_3_conv[0][0]       
__________________________________________________________________________________________________
conv4_block16_preact_bn (BatchN (None, 14, 14, 1024) 4096        conv4_block15_out[0][0]          
__________________________________________________________________________________________________
conv4_block16_preact_relu (Acti (None, 14, 14, 1024) 0           conv4_block16_preact_bn[0][0]    
__________________________________________________________________________________________________
conv4_block16_1_conv (Conv2D)   (None, 14, 14, 256)  262144      conv4_block16_preact_relu[0][0]  
__________________________________________________________________________________________________
conv4_block16_1_bn (BatchNormal (None, 14, 14, 256)  1024        conv4_block16_1_conv[0][0]       
__________________________________________________________________________________________________
conv4_block16_1_relu (Activatio (None, 14, 14, 256)  0           conv4_block16_1_bn[0][0]         
__________________________________________________________________________________________________
conv4_block16_2_pad (ZeroPaddin (None, 16, 16, 256)  0           conv4_block16_1_relu[0][0]       
__________________________________________________________________________________________________
conv4_block16_2_conv (Conv2D)   (None, 14, 14, 256)  589824      conv4_block16_2_pad[0][0]        
__________________________________________________________________________________________________
conv4_block16_2_bn (BatchNormal (None, 14, 14, 256)  1024        conv4_block16_2_conv[0][0]       
__________________________________________________________________________________________________
conv4_block16_2_relu (Activatio (None, 14, 14, 256)  0           conv4_block16_2_bn[0][0]         
__________________________________________________________________________________________________
conv4_block16_3_conv (Conv2D)   (None, 14, 14, 1024) 263168      conv4_block16_2_relu[0][0]       
__________________________________________________________________________________________________
conv4_block16_out (Add)         (None, 14, 14, 1024) 0           conv4_block15_out[0][0]          
                                                                 conv4_block16_3_conv[0][0]       
__________________________________________________________________________________________________
conv4_block17_preact_bn (BatchN (None, 14, 14, 1024) 4096        conv4_block16_out[0][0]          
__________________________________________________________________________________________________
conv4_block17_preact_relu (Acti (None, 14, 14, 1024) 0           conv4_block17_preact_bn[0][0]    
__________________________________________________________________________________________________
conv4_block17_1_conv (Conv2D)   (None, 14, 14, 256)  262144      conv4_block17_preact_relu[0][0]  
__________________________________________________________________________________________________
conv4_block17_1_bn (BatchNormal (None, 14, 14, 256)  1024        conv4_block17_1_conv[0][0]       
__________________________________________________________________________________________________
conv4_block17_1_relu (Activatio (None, 14, 14, 256)  0           conv4_block17_1_bn[0][0]         
__________________________________________________________________________________________________
conv4_block17_2_pad (ZeroPaddin (None, 16, 16, 256)  0           conv4_block17_1_relu[0][0]       
__________________________________________________________________________________________________
conv4_block17_2_conv (Conv2D)   (None, 14, 14, 256)  589824      conv4_block17_2_pad[0][0]        
__________________________________________________________________________________________________
conv4_block17_2_bn (BatchNormal (None, 14, 14, 256)  1024        conv4_block17_2_conv[0][0]       
__________________________________________________________________________________________________
conv4_block17_2_relu (Activatio (None, 14, 14, 256)  0           conv4_block17_2_bn[0][0]         
__________________________________________________________________________________________________
conv4_block17_3_conv (Conv2D)   (None, 14, 14, 1024) 263168      conv4_block17_2_relu[0][0]       
__________________________________________________________________________________________________
conv4_block17_out (Add)         (None, 14, 14, 1024) 0           conv4_block16_out[0][0]          
                                                                 conv4_block17_3_conv[0][0]       
__________________________________________________________________________________________________
conv4_block18_preact_bn (BatchN (None, 14, 14, 1024) 4096        conv4_block17_out[0][0]          
__________________________________________________________________________________________________
conv4_block18_preact_relu (Acti (None, 14, 14, 1024) 0           conv4_block18_preact_bn[0][0]    
__________________________________________________________________________________________________
conv4_block18_1_conv (Conv2D)   (None, 14, 14, 256)  262144      conv4_block18_preact_relu[0][0]  
__________________________________________________________________________________________________
conv4_block18_1_bn (BatchNormal (None, 14, 14, 256)  1024        conv4_block18_1_conv[0][0]       
__________________________________________________________________________________________________
conv4_block18_1_relu (Activatio (None, 14, 14, 256)  0           conv4_block18_1_bn[0][0]         
__________________________________________________________________________________________________
conv4_block18_2_pad (ZeroPaddin (None, 16, 16, 256)  0           conv4_block18_1_relu[0][0]       
__________________________________________________________________________________________________
conv4_block18_2_conv (Conv2D)   (None, 14, 14, 256)  589824      conv4_block18_2_pad[0][0]        
__________________________________________________________________________________________________
conv4_block18_2_bn (BatchNormal (None, 14, 14, 256)  1024        conv4_block18_2_conv[0][0]       
__________________________________________________________________________________________________
conv4_block18_2_relu (Activatio (None, 14, 14, 256)  0           conv4_block18_2_bn[0][0]         
__________________________________________________________________________________________________
conv4_block18_3_conv (Conv2D)   (None, 14, 14, 1024) 263168      conv4_block18_2_relu[0][0]       
__________________________________________________________________________________________________
conv4_block18_out (Add)         (None, 14, 14, 1024) 0           conv4_block17_out[0][0]          
                                                                 conv4_block18_3_conv[0][0]       
__________________________________________________________________________________________________
conv4_block19_preact_bn (BatchN (None, 14, 14, 1024) 4096        conv4_block18_out[0][0]          
__________________________________________________________________________________________________
conv4_block19_preact_relu (Acti (None, 14, 14, 1024) 0           conv4_block19_preact_bn[0][0]    
__________________________________________________________________________________________________
conv4_block19_1_conv (Conv2D)   (None, 14, 14, 256)  262144      conv4_block19_preact_relu[0][0]  
__________________________________________________________________________________________________
conv4_block19_1_bn (BatchNormal (None, 14, 14, 256)  1024        conv4_block19_1_conv[0][0]       
__________________________________________________________________________________________________
conv4_block19_1_relu (Activatio (None, 14, 14, 256)  0           conv4_block19_1_bn[0][0]         
__________________________________________________________________________________________________
conv4_block19_2_pad (ZeroPaddin (None, 16, 16, 256)  0           conv4_block19_1_relu[0][0]       
__________________________________________________________________________________________________
conv4_block19_2_conv (Conv2D)   (None, 14, 14, 256)  589824      conv4_block19_2_pad[0][0]        
__________________________________________________________________________________________________
conv4_block19_2_bn (BatchNormal (None, 14, 14, 256)  1024        conv4_block19_2_conv[0][0]       
__________________________________________________________________________________________________
conv4_block19_2_relu (Activatio (None, 14, 14, 256)  0           conv4_block19_2_bn[0][0]         
__________________________________________________________________________________________________
conv4_block19_3_conv (Conv2D)   (None, 14, 14, 1024) 263168      conv4_block19_2_relu[0][0]       
__________________________________________________________________________________________________
conv4_block19_out (Add)         (None, 14, 14, 1024) 0           conv4_block18_out[0][0]          
                                                                 conv4_block19_3_conv[0][0]       
__________________________________________________________________________________________________
conv4_block20_preact_bn (BatchN (None, 14, 14, 1024) 4096        conv4_block19_out[0][0]          
__________________________________________________________________________________________________
conv4_block20_preact_relu (Acti (None, 14, 14, 1024) 0           conv4_block20_preact_bn[0][0]    
__________________________________________________________________________________________________
conv4_block20_1_conv (Conv2D)   (None, 14, 14, 256)  262144      conv4_block20_preact_relu[0][0]  
__________________________________________________________________________________________________
conv4_block20_1_bn (BatchNormal (None, 14, 14, 256)  1024        conv4_block20_1_conv[0][0]       
__________________________________________________________________________________________________
conv4_block20_1_relu (Activatio (None, 14, 14, 256)  0           conv4_block20_1_bn[0][0]         
__________________________________________________________________________________________________
conv4_block20_2_pad (ZeroPaddin (None, 16, 16, 256)  0           conv4_block20_1_relu[0][0]       
__________________________________________________________________________________________________
conv4_block20_2_conv (Conv2D)   (None, 14, 14, 256)  589824      conv4_block20_2_pad[0][0]        
__________________________________________________________________________________________________
conv4_block20_2_bn (BatchNormal (None, 14, 14, 256)  1024        conv4_block20_2_conv[0][0]       
__________________________________________________________________________________________________
conv4_block20_2_relu (Activatio (None, 14, 14, 256)  0           conv4_block20_2_bn[0][0]         
__________________________________________________________________________________________________
conv4_block20_3_conv (Conv2D)   (None, 14, 14, 1024) 263168      conv4_block20_2_relu[0][0]       
__________________________________________________________________________________________________
conv4_block20_out (Add)         (None, 14, 14, 1024) 0           conv4_block19_out[0][0]          
                                                                 conv4_block20_3_conv[0][0]       
__________________________________________________________________________________________________
conv4_block21_preact_bn (BatchN (None, 14, 14, 1024) 4096        conv4_block20_out[0][0]          
__________________________________________________________________________________________________
conv4_block21_preact_relu (Acti (None, 14, 14, 1024) 0           conv4_block21_preact_bn[0][0]    
__________________________________________________________________________________________________
conv4_block21_1_conv (Conv2D)   (None, 14, 14, 256)  262144      conv4_block21_preact_relu[0][0]  
__________________________________________________________________________________________________
conv4_block21_1_bn (BatchNormal (None, 14, 14, 256)  1024        conv4_block21_1_conv[0][0]       
__________________________________________________________________________________________________
conv4_block21_1_relu (Activatio (None, 14, 14, 256)  0           conv4_block21_1_bn[0][0]         
__________________________________________________________________________________________________
conv4_block21_2_pad (ZeroPaddin (None, 16, 16, 256)  0           conv4_block21_1_relu[0][0]       
__________________________________________________________________________________________________
conv4_block21_2_conv (Conv2D)   (None, 14, 14, 256)  589824      conv4_block21_2_pad[0][0]        
__________________________________________________________________________________________________
conv4_block21_2_bn (BatchNormal (None, 14, 14, 256)  1024        conv4_block21_2_conv[0][0]       
__________________________________________________________________________________________________
conv4_block21_2_relu (Activatio (None, 14, 14, 256)  0           conv4_block21_2_bn[0][0]         
__________________________________________________________________________________________________
conv4_block21_3_conv (Conv2D)   (None, 14, 14, 1024) 263168      conv4_block21_2_relu[0][0]       
__________________________________________________________________________________________________
conv4_block21_out (Add)         (None, 14, 14, 1024) 0           conv4_block20_out[0][0]          
                                                                 conv4_block21_3_conv[0][0]       
__________________________________________________________________________________________________
conv4_block22_preact_bn (BatchN (None, 14, 14, 1024) 4096        conv4_block21_out[0][0]          
__________________________________________________________________________________________________
conv4_block22_preact_relu (Acti (None, 14, 14, 1024) 0           conv4_block22_preact_bn[0][0]    
__________________________________________________________________________________________________
conv4_block22_1_conv (Conv2D)   (None, 14, 14, 256)  262144      conv4_block22_preact_relu[0][0]  
__________________________________________________________________________________________________
conv4_block22_1_bn (BatchNormal (None, 14, 14, 256)  1024        conv4_block22_1_conv[0][0]       
__________________________________________________________________________________________________
conv4_block22_1_relu (Activatio (None, 14, 14, 256)  0           conv4_block22_1_bn[0][0]         
__________________________________________________________________________________________________
conv4_block22_2_pad (ZeroPaddin (None, 16, 16, 256)  0           conv4_block22_1_relu[0][0]       
__________________________________________________________________________________________________
conv4_block22_2_conv (Conv2D)   (None, 14, 14, 256)  589824      conv4_block22_2_pad[0][0]        
__________________________________________________________________________________________________
conv4_block22_2_bn (BatchNormal (None, 14, 14, 256)  1024        conv4_block22_2_conv[0][0]       
__________________________________________________________________________________________________
conv4_block22_2_relu (Activatio (None, 14, 14, 256)  0           conv4_block22_2_bn[0][0]         
__________________________________________________________________________________________________
conv4_block22_3_conv (Conv2D)   (None, 14, 14, 1024) 263168      conv4_block22_2_relu[0][0]       
__________________________________________________________________________________________________
conv4_block22_out (Add)         (None, 14, 14, 1024) 0           conv4_block21_out[0][0]          
                                                                 conv4_block22_3_conv[0][0]       
__________________________________________________________________________________________________
conv4_block23_preact_bn (BatchN (None, 14, 14, 1024) 4096        conv4_block22_out[0][0]          
__________________________________________________________________________________________________
conv4_block23_preact_relu (Acti (None, 14, 14, 1024) 0           conv4_block23_preact_bn[0][0]    
__________________________________________________________________________________________________
conv4_block23_1_conv (Conv2D)   (None, 14, 14, 256)  262144      conv4_block23_preact_relu[0][0]  
__________________________________________________________________________________________________
conv4_block23_1_bn (BatchNormal (None, 14, 14, 256)  1024        conv4_block23_1_conv[0][0]       
__________________________________________________________________________________________________
conv4_block23_1_relu (Activatio (None, 14, 14, 256)  0           conv4_block23_1_bn[0][0]         
__________________________________________________________________________________________________
conv4_block23_2_pad (ZeroPaddin (None, 16, 16, 256)  0           conv4_block23_1_relu[0][0]       
__________________________________________________________________________________________________
conv4_block23_2_conv (Conv2D)   (None, 14, 14, 256)  589824      conv4_block23_2_pad[0][0]        
__________________________________________________________________________________________________
conv4_block23_2_bn (BatchNormal (None, 14, 14, 256)  1024        conv4_block23_2_conv[0][0]       
__________________________________________________________________________________________________
conv4_block23_2_relu (Activatio (None, 14, 14, 256)  0           conv4_block23_2_bn[0][0]         
__________________________________________________________________________________________________
conv4_block23_3_conv (Conv2D)   (None, 14, 14, 1024) 263168      conv4_block23_2_relu[0][0]       
__________________________________________________________________________________________________
conv4_block23_out (Add)         (None, 14, 14, 1024) 0           conv4_block22_out[0][0]          
                                                                 conv4_block23_3_conv[0][0]       
__________________________________________________________________________________________________
conv4_block24_preact_bn (BatchN (None, 14, 14, 1024) 4096        conv4_block23_out[0][0]          
__________________________________________________________________________________________________
conv4_block24_preact_relu (Acti (None, 14, 14, 1024) 0           conv4_block24_preact_bn[0][0]    
__________________________________________________________________________________________________
conv4_block24_1_conv (Conv2D)   (None, 14, 14, 256)  262144      conv4_block24_preact_relu[0][0]  
__________________________________________________________________________________________________
conv4_block24_1_bn (BatchNormal (None, 14, 14, 256)  1024        conv4_block24_1_conv[0][0]       
__________________________________________________________________________________________________
conv4_block24_1_relu (Activatio (None, 14, 14, 256)  0           conv4_block24_1_bn[0][0]         
__________________________________________________________________________________________________
conv4_block24_2_pad (ZeroPaddin (None, 16, 16, 256)  0           conv4_block24_1_relu[0][0]       
__________________________________________________________________________________________________
conv4_block24_2_conv (Conv2D)   (None, 14, 14, 256)  589824      conv4_block24_2_pad[0][0]        
__________________________________________________________________________________________________
conv4_block24_2_bn (BatchNormal (None, 14, 14, 256)  1024        conv4_block24_2_conv[0][0]       
__________________________________________________________________________________________________
conv4_block24_2_relu (Activatio (None, 14, 14, 256)  0           conv4_block24_2_bn[0][0]         
__________________________________________________________________________________________________
conv4_block24_3_conv (Conv2D)   (None, 14, 14, 1024) 263168      conv4_block24_2_relu[0][0]       
__________________________________________________________________________________________________
conv4_block24_out (Add)         (None, 14, 14, 1024) 0           conv4_block23_out[0][0]          
                                                                 conv4_block24_3_conv[0][0]       
__________________________________________________________________________________________________
conv4_block25_preact_bn (BatchN (None, 14, 14, 1024) 4096        conv4_block24_out[0][0]          
__________________________________________________________________________________________________
conv4_block25_preact_relu (Acti (None, 14, 14, 1024) 0           conv4_block25_preact_bn[0][0]    
__________________________________________________________________________________________________
conv4_block25_1_conv (Conv2D)   (None, 14, 14, 256)  262144      conv4_block25_preact_relu[0][0]  
__________________________________________________________________________________________________
conv4_block25_1_bn (BatchNormal (None, 14, 14, 256)  1024        conv4_block25_1_conv[0][0]       
__________________________________________________________________________________________________
conv4_block25_1_relu (Activatio (None, 14, 14, 256)  0           conv4_block25_1_bn[0][0]         
__________________________________________________________________________________________________
conv4_block25_2_pad (ZeroPaddin (None, 16, 16, 256)  0           conv4_block25_1_relu[0][0]       
__________________________________________________________________________________________________
conv4_block25_2_conv (Conv2D)   (None, 14, 14, 256)  589824      conv4_block25_2_pad[0][0]        
__________________________________________________________________________________________________
conv4_block25_2_bn (BatchNormal (None, 14, 14, 256)  1024        conv4_block25_2_conv[0][0]       
__________________________________________________________________________________________________
conv4_block25_2_relu (Activatio (None, 14, 14, 256)  0           conv4_block25_2_bn[0][0]         
__________________________________________________________________________________________________
conv4_block25_3_conv (Conv2D)   (None, 14, 14, 1024) 263168      conv4_block25_2_relu[0][0]       
__________________________________________________________________________________________________
conv4_block25_out (Add)         (None, 14, 14, 1024) 0           conv4_block24_out[0][0]          
                                                                 conv4_block25_3_conv[0][0]       
__________________________________________________________________________________________________
conv4_block26_preact_bn (BatchN (None, 14, 14, 1024) 4096        conv4_block25_out[0][0]          
__________________________________________________________________________________________________
conv4_block26_preact_relu (Acti (None, 14, 14, 1024) 0           conv4_block26_preact_bn[0][0]    
__________________________________________________________________________________________________
conv4_block26_1_conv (Conv2D)   (None, 14, 14, 256)  262144      conv4_block26_preact_relu[0][0]  
__________________________________________________________________________________________________
conv4_block26_1_bn (BatchNormal (None, 14, 14, 256)  1024        conv4_block26_1_conv[0][0]       
__________________________________________________________________________________________________
conv4_block26_1_relu (Activatio (None, 14, 14, 256)  0           conv4_block26_1_bn[0][0]         
__________________________________________________________________________________________________
conv4_block26_2_pad (ZeroPaddin (None, 16, 16, 256)  0           conv4_block26_1_relu[0][0]       
__________________________________________________________________________________________________
conv4_block26_2_conv (Conv2D)   (None, 14, 14, 256)  589824      conv4_block26_2_pad[0][0]        
__________________________________________________________________________________________________
conv4_block26_2_bn (BatchNormal (None, 14, 14, 256)  1024        conv4_block26_2_conv[0][0]       
__________________________________________________________________________________________________
conv4_block26_2_relu (Activatio (None, 14, 14, 256)  0           conv4_block26_2_bn[0][0]         
__________________________________________________________________________________________________
conv4_block26_3_conv (Conv2D)   (None, 14, 14, 1024) 263168      conv4_block26_2_relu[0][0]       
__________________________________________________________________________________________________
conv4_block26_out (Add)         (None, 14, 14, 1024) 0           conv4_block25_out[0][0]          
                                                                 conv4_block26_3_conv[0][0]       
__________________________________________________________________________________________________
conv4_block27_preact_bn (BatchN (None, 14, 14, 1024) 4096        conv4_block26_out[0][0]          
__________________________________________________________________________________________________
conv4_block27_preact_relu (Acti (None, 14, 14, 1024) 0           conv4_block27_preact_bn[0][0]    
__________________________________________________________________________________________________
conv4_block27_1_conv (Conv2D)   (None, 14, 14, 256)  262144      conv4_block27_preact_relu[0][0]  
__________________________________________________________________________________________________
conv4_block27_1_bn (BatchNormal (None, 14, 14, 256)  1024        conv4_block27_1_conv[0][0]       
__________________________________________________________________________________________________
conv4_block27_1_relu (Activatio (None, 14, 14, 256)  0           conv4_block27_1_bn[0][0]         
__________________________________________________________________________________________________
conv4_block27_2_pad (ZeroPaddin (None, 16, 16, 256)  0           conv4_block27_1_relu[0][0]       
__________________________________________________________________________________________________
conv4_block27_2_conv (Conv2D)   (None, 14, 14, 256)  589824      conv4_block27_2_pad[0][0]        
__________________________________________________________________________________________________
conv4_block27_2_bn (BatchNormal (None, 14, 14, 256)  1024        conv4_block27_2_conv[0][0]       
__________________________________________________________________________________________________
conv4_block27_2_relu (Activatio (None, 14, 14, 256)  0           conv4_block27_2_bn[0][0]         
__________________________________________________________________________________________________
conv4_block27_3_conv (Conv2D)   (None, 14, 14, 1024) 263168      conv4_block27_2_relu[0][0]       
__________________________________________________________________________________________________
conv4_block27_out (Add)         (None, 14, 14, 1024) 0           conv4_block26_out[0][0]          
                                                                 conv4_block27_3_conv[0][0]       
__________________________________________________________________________________________________
conv4_block28_preact_bn (BatchN (None, 14, 14, 1024) 4096        conv4_block27_out[0][0]          
__________________________________________________________________________________________________
conv4_block28_preact_relu (Acti (None, 14, 14, 1024) 0           conv4_block28_preact_bn[0][0]    
__________________________________________________________________________________________________
conv4_block28_1_conv (Conv2D)   (None, 14, 14, 256)  262144      conv4_block28_preact_relu[0][0]  
__________________________________________________________________________________________________
conv4_block28_1_bn (BatchNormal (None, 14, 14, 256)  1024        conv4_block28_1_conv[0][0]       
__________________________________________________________________________________________________
conv4_block28_1_relu (Activatio (None, 14, 14, 256)  0           conv4_block28_1_bn[0][0]         
__________________________________________________________________________________________________
conv4_block28_2_pad (ZeroPaddin (None, 16, 16, 256)  0           conv4_block28_1_relu[0][0]       
__________________________________________________________________________________________________
conv4_block28_2_conv (Conv2D)   (None, 14, 14, 256)  589824      conv4_block28_2_pad[0][0]        
__________________________________________________________________________________________________
conv4_block28_2_bn (BatchNormal (None, 14, 14, 256)  1024        conv4_block28_2_conv[0][0]       
__________________________________________________________________________________________________
conv4_block28_2_relu (Activatio (None, 14, 14, 256)  0           conv4_block28_2_bn[0][0]         
__________________________________________________________________________________________________
conv4_block28_3_conv (Conv2D)   (None, 14, 14, 1024) 263168      conv4_block28_2_relu[0][0]       
__________________________________________________________________________________________________
conv4_block28_out (Add)         (None, 14, 14, 1024) 0           conv4_block27_out[0][0]          
                                                                 conv4_block28_3_conv[0][0]       
__________________________________________________________________________________________________
conv4_block29_preact_bn (BatchN (None, 14, 14, 1024) 4096        conv4_block28_out[0][0]          
__________________________________________________________________________________________________
conv4_block29_preact_relu (Acti (None, 14, 14, 1024) 0           conv4_block29_preact_bn[0][0]    
__________________________________________________________________________________________________
conv4_block29_1_conv (Conv2D)   (None, 14, 14, 256)  262144      conv4_block29_preact_relu[0][0]  
__________________________________________________________________________________________________
conv4_block29_1_bn (BatchNormal (None, 14, 14, 256)  1024        conv4_block29_1_conv[0][0]       
__________________________________________________________________________________________________
conv4_block29_1_relu (Activatio (None, 14, 14, 256)  0           conv4_block29_1_bn[0][0]         
__________________________________________________________________________________________________
conv4_block29_2_pad (ZeroPaddin (None, 16, 16, 256)  0           conv4_block29_1_relu[0][0]       
__________________________________________________________________________________________________
conv4_block29_2_conv (Conv2D)   (None, 14, 14, 256)  589824      conv4_block29_2_pad[0][0]        
__________________________________________________________________________________________________
conv4_block29_2_bn (BatchNormal (None, 14, 14, 256)  1024        conv4_block29_2_conv[0][0]       
__________________________________________________________________________________________________
conv4_block29_2_relu (Activatio (None, 14, 14, 256)  0           conv4_block29_2_bn[0][0]         
__________________________________________________________________________________________________
conv4_block29_3_conv (Conv2D)   (None, 14, 14, 1024) 263168      conv4_block29_2_relu[0][0]       
__________________________________________________________________________________________________
conv4_block29_out (Add)         (None, 14, 14, 1024) 0           conv4_block28_out[0][0]          
                                                                 conv4_block29_3_conv[0][0]       
__________________________________________________________________________________________________
conv4_block30_preact_bn (BatchN (None, 14, 14, 1024) 4096        conv4_block29_out[0][0]          
__________________________________________________________________________________________________
conv4_block30_preact_relu (Acti (None, 14, 14, 1024) 0           conv4_block30_preact_bn[0][0]    
__________________________________________________________________________________________________
conv4_block30_1_conv (Conv2D)   (None, 14, 14, 256)  262144      conv4_block30_preact_relu[0][0]  
__________________________________________________________________________________________________
conv4_block30_1_bn (BatchNormal (None, 14, 14, 256)  1024        conv4_block30_1_conv[0][0]       
__________________________________________________________________________________________________
conv4_block30_1_relu (Activatio (None, 14, 14, 256)  0           conv4_block30_1_bn[0][0]         
__________________________________________________________________________________________________
conv4_block30_2_pad (ZeroPaddin (None, 16, 16, 256)  0           conv4_block30_1_relu[0][0]       
__________________________________________________________________________________________________
conv4_block30_2_conv (Conv2D)   (None, 14, 14, 256)  589824      conv4_block30_2_pad[0][0]        
__________________________________________________________________________________________________
conv4_block30_2_bn (BatchNormal (None, 14, 14, 256)  1024        conv4_block30_2_conv[0][0]       
__________________________________________________________________________________________________
conv4_block30_2_relu (Activatio (None, 14, 14, 256)  0           conv4_block30_2_bn[0][0]         
__________________________________________________________________________________________________
conv4_block30_3_conv (Conv2D)   (None, 14, 14, 1024) 263168      conv4_block30_2_relu[0][0]       
__________________________________________________________________________________________________
conv4_block30_out (Add)         (None, 14, 14, 1024) 0           conv4_block29_out[0][0]          
                                                                 conv4_block30_3_conv[0][0]       
__________________________________________________________________________________________________
conv4_block31_preact_bn (BatchN (None, 14, 14, 1024) 4096        conv4_block30_out[0][0]          
__________________________________________________________________________________________________
conv4_block31_preact_relu (Acti (None, 14, 14, 1024) 0           conv4_block31_preact_bn[0][0]    
__________________________________________________________________________________________________
conv4_block31_1_conv (Conv2D)   (None, 14, 14, 256)  262144      conv4_block31_preact_relu[0][0]  
__________________________________________________________________________________________________
conv4_block31_1_bn (BatchNormal (None, 14, 14, 256)  1024        conv4_block31_1_conv[0][0]       
__________________________________________________________________________________________________
conv4_block31_1_relu (Activatio (None, 14, 14, 256)  0           conv4_block31_1_bn[0][0]         
__________________________________________________________________________________________________
conv4_block31_2_pad (ZeroPaddin (None, 16, 16, 256)  0           conv4_block31_1_relu[0][0]       
__________________________________________________________________________________________________
conv4_block31_2_conv (Conv2D)   (None, 14, 14, 256)  589824      conv4_block31_2_pad[0][0]        
__________________________________________________________________________________________________
conv4_block31_2_bn (BatchNormal (None, 14, 14, 256)  1024        conv4_block31_2_conv[0][0]       
__________________________________________________________________________________________________
conv4_block31_2_relu (Activatio (None, 14, 14, 256)  0           conv4_block31_2_bn[0][0]         
__________________________________________________________________________________________________
conv4_block31_3_conv (Conv2D)   (None, 14, 14, 1024) 263168      conv4_block31_2_relu[0][0]       
__________________________________________________________________________________________________
conv4_block31_out (Add)         (None, 14, 14, 1024) 0           conv4_block30_out[0][0]          
                                                                 conv4_block31_3_conv[0][0]       
__________________________________________________________________________________________________
conv4_block32_preact_bn (BatchN (None, 14, 14, 1024) 4096        conv4_block31_out[0][0]          
__________________________________________________________________________________________________
conv4_block32_preact_relu (Acti (None, 14, 14, 1024) 0           conv4_block32_preact_bn[0][0]    
__________________________________________________________________________________________________
conv4_block32_1_conv (Conv2D)   (None, 14, 14, 256)  262144      conv4_block32_preact_relu[0][0]  
__________________________________________________________________________________________________
conv4_block32_1_bn (BatchNormal (None, 14, 14, 256)  1024        conv4_block32_1_conv[0][0]       
__________________________________________________________________________________________________
conv4_block32_1_relu (Activatio (None, 14, 14, 256)  0           conv4_block32_1_bn[0][0]         
__________________________________________________________________________________________________
conv4_block32_2_pad (ZeroPaddin (None, 16, 16, 256)  0           conv4_block32_1_relu[0][0]       
__________________________________________________________________________________________________
conv4_block32_2_conv (Conv2D)   (None, 14, 14, 256)  589824      conv4_block32_2_pad[0][0]        
__________________________________________________________________________________________________
conv4_block32_2_bn (BatchNormal (None, 14, 14, 256)  1024        conv4_block32_2_conv[0][0]       
__________________________________________________________________________________________________
conv4_block32_2_relu (Activatio (None, 14, 14, 256)  0           conv4_block32_2_bn[0][0]         
__________________________________________________________________________________________________
conv4_block32_3_conv (Conv2D)   (None, 14, 14, 1024) 263168      conv4_block32_2_relu[0][0]       
__________________________________________________________________________________________________
conv4_block32_out (Add)         (None, 14, 14, 1024) 0           conv4_block31_out[0][0]          
                                                                 conv4_block32_3_conv[0][0]       
__________________________________________________________________________________________________
conv4_block33_preact_bn (BatchN (None, 14, 14, 1024) 4096        conv4_block32_out[0][0]          
__________________________________________________________________________________________________
conv4_block33_preact_relu (Acti (None, 14, 14, 1024) 0           conv4_block33_preact_bn[0][0]    
__________________________________________________________________________________________________
conv4_block33_1_conv (Conv2D)   (None, 14, 14, 256)  262144      conv4_block33_preact_relu[0][0]  
__________________________________________________________________________________________________
conv4_block33_1_bn (BatchNormal (None, 14, 14, 256)  1024        conv4_block33_1_conv[0][0]       
__________________________________________________________________________________________________
conv4_block33_1_relu (Activatio (None, 14, 14, 256)  0           conv4_block33_1_bn[0][0]         
__________________________________________________________________________________________________
conv4_block33_2_pad (ZeroPaddin (None, 16, 16, 256)  0           conv4_block33_1_relu[0][0]       
__________________________________________________________________________________________________
conv4_block33_2_conv (Conv2D)   (None, 14, 14, 256)  589824      conv4_block33_2_pad[0][0]        
__________________________________________________________________________________________________
conv4_block33_2_bn (BatchNormal (None, 14, 14, 256)  1024        conv4_block33_2_conv[0][0]       
__________________________________________________________________________________________________
conv4_block33_2_relu (Activatio (None, 14, 14, 256)  0           conv4_block33_2_bn[0][0]         
__________________________________________________________________________________________________
conv4_block33_3_conv (Conv2D)   (None, 14, 14, 1024) 263168      conv4_block33_2_relu[0][0]       
__________________________________________________________________________________________________
conv4_block33_out (Add)         (None, 14, 14, 1024) 0           conv4_block32_out[0][0]          
                                                                 conv4_block33_3_conv[0][0]       
__________________________________________________________________________________________________
conv4_block34_preact_bn (BatchN (None, 14, 14, 1024) 4096        conv4_block33_out[0][0]          
__________________________________________________________________________________________________
conv4_block34_preact_relu (Acti (None, 14, 14, 1024) 0           conv4_block34_preact_bn[0][0]    
__________________________________________________________________________________________________
conv4_block34_1_conv (Conv2D)   (None, 14, 14, 256)  262144      conv4_block34_preact_relu[0][0]  
__________________________________________________________________________________________________
conv4_block34_1_bn (BatchNormal (None, 14, 14, 256)  1024        conv4_block34_1_conv[0][0]       
__________________________________________________________________________________________________
conv4_block34_1_relu (Activatio (None, 14, 14, 256)  0           conv4_block34_1_bn[0][0]         
__________________________________________________________________________________________________
conv4_block34_2_pad (ZeroPaddin (None, 16, 16, 256)  0           conv4_block34_1_relu[0][0]       
__________________________________________________________________________________________________
conv4_block34_2_conv (Conv2D)   (None, 14, 14, 256)  589824      conv4_block34_2_pad[0][0]        
__________________________________________________________________________________________________
conv4_block34_2_bn (BatchNormal (None, 14, 14, 256)  1024        conv4_block34_2_conv[0][0]       
__________________________________________________________________________________________________
conv4_block34_2_relu (Activatio (None, 14, 14, 256)  0           conv4_block34_2_bn[0][0]         
__________________________________________________________________________________________________
conv4_block34_3_conv (Conv2D)   (None, 14, 14, 1024) 263168      conv4_block34_2_relu[0][0]       
__________________________________________________________________________________________________
conv4_block34_out (Add)         (None, 14, 14, 1024) 0           conv4_block33_out[0][0]          
                                                                 conv4_block34_3_conv[0][0]       
__________________________________________________________________________________________________
conv4_block35_preact_bn (BatchN (None, 14, 14, 1024) 4096        conv4_block34_out[0][0]          
__________________________________________________________________________________________________
conv4_block35_preact_relu (Acti (None, 14, 14, 1024) 0           conv4_block35_preact_bn[0][0]    
__________________________________________________________________________________________________
conv4_block35_1_conv (Conv2D)   (None, 14, 14, 256)  262144      conv4_block35_preact_relu[0][0]  
__________________________________________________________________________________________________
conv4_block35_1_bn (BatchNormal (None, 14, 14, 256)  1024        conv4_block35_1_conv[0][0]       
__________________________________________________________________________________________________
conv4_block35_1_relu (Activatio (None, 14, 14, 256)  0           conv4_block35_1_bn[0][0]         
__________________________________________________________________________________________________
conv4_block35_2_pad (ZeroPaddin (None, 16, 16, 256)  0           conv4_block35_1_relu[0][0]       
__________________________________________________________________________________________________
conv4_block35_2_conv (Conv2D)   (None, 14, 14, 256)  589824      conv4_block35_2_pad[0][0]        
__________________________________________________________________________________________________
conv4_block35_2_bn (BatchNormal (None, 14, 14, 256)  1024        conv4_block35_2_conv[0][0]       
__________________________________________________________________________________________________
conv4_block35_2_relu (Activatio (None, 14, 14, 256)  0           conv4_block35_2_bn[0][0]         
__________________________________________________________________________________________________
conv4_block35_3_conv (Conv2D)   (None, 14, 14, 1024) 263168      conv4_block35_2_relu[0][0]       
__________________________________________________________________________________________________
conv4_block35_out (Add)         (None, 14, 14, 1024) 0           conv4_block34_out[0][0]          
                                                                 conv4_block35_3_conv[0][0]       
__________________________________________________________________________________________________
conv4_block36_preact_bn (BatchN (None, 14, 14, 1024) 4096        conv4_block35_out[0][0]          
__________________________________________________________________________________________________
conv4_block36_preact_relu (Acti (None, 14, 14, 1024) 0           conv4_block36_preact_bn[0][0]    
__________________________________________________________________________________________________
conv4_block36_1_conv (Conv2D)   (None, 14, 14, 256)  262144      conv4_block36_preact_relu[0][0]  
__________________________________________________________________________________________________
conv4_block36_1_bn (BatchNormal (None, 14, 14, 256)  1024        conv4_block36_1_conv[0][0]       
__________________________________________________________________________________________________
conv4_block36_1_relu (Activatio (None, 14, 14, 256)  0           conv4_block36_1_bn[0][0]         
__________________________________________________________________________________________________
conv4_block36_2_pad (ZeroPaddin (None, 16, 16, 256)  0           conv4_block36_1_relu[0][0]       
__________________________________________________________________________________________________
conv4_block36_2_conv (Conv2D)   (None, 7, 7, 256)    589824      conv4_block36_2_pad[0][0]        
__________________________________________________________________________________________________
conv4_block36_2_bn (BatchNormal (None, 7, 7, 256)    1024        conv4_block36_2_conv[0][0]       
__________________________________________________________________________________________________
conv4_block36_2_relu (Activatio (None, 7, 7, 256)    0           conv4_block36_2_bn[0][0]         
__________________________________________________________________________________________________
max_pooling2d_5 (MaxPooling2D)  (None, 7, 7, 1024)   0           conv4_block35_out[0][0]          
__________________________________________________________________________________________________
conv4_block36_3_conv (Conv2D)   (None, 7, 7, 1024)   263168      conv4_block36_2_relu[0][0]       
__________________________________________________________________________________________________
conv4_block36_out (Add)         (None, 7, 7, 1024)   0           max_pooling2d_5[0][0]            
                                                                 conv4_block36_3_conv[0][0]       
__________________________________________________________________________________________________
conv5_block1_preact_bn (BatchNo (None, 7, 7, 1024)   4096        conv4_block36_out[0][0]          
__________________________________________________________________________________________________
conv5_block1_preact_relu (Activ (None, 7, 7, 1024)   0           conv5_block1_preact_bn[0][0]     
__________________________________________________________________________________________________
conv5_block1_1_conv (Conv2D)    (None, 7, 7, 512)    524288      conv5_block1_preact_relu[0][0]   
__________________________________________________________________________________________________
conv5_block1_1_bn (BatchNormali (None, 7, 7, 512)    2048        conv5_block1_1_conv[0][0]        
__________________________________________________________________________________________________
conv5_block1_1_relu (Activation (None, 7, 7, 512)    0           conv5_block1_1_bn[0][0]          
__________________________________________________________________________________________________
conv5_block1_2_pad (ZeroPadding (None, 9, 9, 512)    0           conv5_block1_1_relu[0][0]        
__________________________________________________________________________________________________
conv5_block1_2_conv (Conv2D)    (None, 7, 7, 512)    2359296     conv5_block1_2_pad[0][0]         
__________________________________________________________________________________________________
conv5_block1_2_bn (BatchNormali (None, 7, 7, 512)    2048        conv5_block1_2_conv[0][0]        
__________________________________________________________________________________________________
conv5_block1_2_relu (Activation (None, 7, 7, 512)    0           conv5_block1_2_bn[0][0]          
__________________________________________________________________________________________________
conv5_block1_0_conv (Conv2D)    (None, 7, 7, 2048)   2099200     conv5_block1_preact_relu[0][0]   
__________________________________________________________________________________________________
conv5_block1_3_conv (Conv2D)    (None, 7, 7, 2048)   1050624     conv5_block1_2_relu[0][0]        
__________________________________________________________________________________________________
conv5_block1_out (Add)          (None, 7, 7, 2048)   0           conv5_block1_0_conv[0][0]        
                                                                 conv5_block1_3_conv[0][0]        
__________________________________________________________________________________________________
conv5_block2_preact_bn (BatchNo (None, 7, 7, 2048)   8192        conv5_block1_out[0][0]           
__________________________________________________________________________________________________
conv5_block2_preact_relu (Activ (None, 7, 7, 2048)   0           conv5_block2_preact_bn[0][0]     
__________________________________________________________________________________________________
conv5_block2_1_conv (Conv2D)    (None, 7, 7, 512)    1048576     conv5_block2_preact_relu[0][0]   
__________________________________________________________________________________________________
conv5_block2_1_bn (BatchNormali (None, 7, 7, 512)    2048        conv5_block2_1_conv[0][0]        
__________________________________________________________________________________________________
conv5_block2_1_relu (Activation (None, 7, 7, 512)    0           conv5_block2_1_bn[0][0]          
__________________________________________________________________________________________________
conv5_block2_2_pad (ZeroPadding (None, 9, 9, 512)    0           conv5_block2_1_relu[0][0]        
__________________________________________________________________________________________________
conv5_block2_2_conv (Conv2D)    (None, 7, 7, 512)    2359296     conv5_block2_2_pad[0][0]         
__________________________________________________________________________________________________
conv5_block2_2_bn (BatchNormali (None, 7, 7, 512)    2048        conv5_block2_2_conv[0][0]        
__________________________________________________________________________________________________
conv5_block2_2_relu (Activation (None, 7, 7, 512)    0           conv5_block2_2_bn[0][0]          
__________________________________________________________________________________________________
conv5_block2_3_conv (Conv2D)    (None, 7, 7, 2048)   1050624     conv5_block2_2_relu[0][0]        
__________________________________________________________________________________________________
conv5_block2_out (Add)          (None, 7, 7, 2048)   0           conv5_block1_out[0][0]           
                                                                 conv5_block2_3_conv[0][0]        
__________________________________________________________________________________________________
conv5_block3_preact_bn (BatchNo (None, 7, 7, 2048)   8192        conv5_block2_out[0][0]           
__________________________________________________________________________________________________
conv5_block3_preact_relu (Activ (None, 7, 7, 2048)   0           conv5_block3_preact_bn[0][0]     
__________________________________________________________________________________________________
conv5_block3_1_conv (Conv2D)    (None, 7, 7, 512)    1048576     conv5_block3_preact_relu[0][0]   
__________________________________________________________________________________________________
conv5_block3_1_bn (BatchNormali (None, 7, 7, 512)    2048        conv5_block3_1_conv[0][0]        
__________________________________________________________________________________________________
conv5_block3_1_relu (Activation (None, 7, 7, 512)    0           conv5_block3_1_bn[0][0]          
__________________________________________________________________________________________________
conv5_block3_2_pad (ZeroPadding (None, 9, 9, 512)    0           conv5_block3_1_relu[0][0]        
__________________________________________________________________________________________________
conv5_block3_2_conv (Conv2D)    (None, 7, 7, 512)    2359296     conv5_block3_2_pad[0][0]         
__________________________________________________________________________________________________
conv5_block3_2_bn (BatchNormali (None, 7, 7, 512)    2048        conv5_block3_2_conv[0][0]        
__________________________________________________________________________________________________
conv5_block3_2_relu (Activation (None, 7, 7, 512)    0           conv5_block3_2_bn[0][0]          
__________________________________________________________________________________________________
conv5_block3_3_conv (Conv2D)    (None, 7, 7, 2048)   1050624     conv5_block3_2_relu[0][0]        
__________________________________________________________________________________________________
conv5_block3_out (Add)          (None, 7, 7, 2048)   0           conv5_block2_out[0][0]           
                                                                 conv5_block3_3_conv[0][0]        
__________________________________________________________________________________________________
post_bn (BatchNormalization)    (None, 7, 7, 2048)   8192        conv5_block3_out[0][0]           
__________________________________________________________________________________________________
post_relu (Activation)          (None, 7, 7, 2048)   0           post_bn[0][0]                    
__________________________________________________________________________________________________
avg_pool (GlobalAveragePooling2 (None, 2048)         0           post_relu[0][0]                  
__________________________________________________________________________________________________
predictions (Dense)             (None, 1000)         2049000     avg_pool[0][0]                   
==================================================================================================
Total params: 60,380,648
Trainable params: 60,236,904
Non-trainable params: 143,744
__________________________________________________________________________________________________

inception = tf.keras.applications.InceptionV3() 
inception.summary() # concatenate

Model: "inception_v3"
__________________________________________________________________________________________________
Layer (type)                    Output Shape         Param #     Connected to                     
==================================================================================================
input_3 (InputLayer)            [(None, 299, 299, 3) 0                                            
__________________________________________________________________________________________________
conv2d (Conv2D)                 (None, 149, 149, 32) 864         input_3[0][0]                    
__________________________________________________________________________________________________
batch_normalization (BatchNorma (None, 149, 149, 32) 96          conv2d[0][0]                     
__________________________________________________________________________________________________
activation (Activation)         (None, 149, 149, 32) 0           batch_normalization[0][0]        
__________________________________________________________________________________________________
conv2d_1 (Conv2D)               (None, 147, 147, 32) 9216        activation[0][0]                 
__________________________________________________________________________________________________
batch_normalization_1 (BatchNor (None, 147, 147, 32) 96          conv2d_1[0][0]                   
__________________________________________________________________________________________________
activation_1 (Activation)       (None, 147, 147, 32) 0           batch_normalization_1[0][0]      
__________________________________________________________________________________________________
conv2d_2 (Conv2D)               (None, 147, 147, 64) 18432       activation_1[0][0]               
__________________________________________________________________________________________________
batch_normalization_2 (BatchNor (None, 147, 147, 64) 192         conv2d_2[0][0]                   
__________________________________________________________________________________________________
activation_2 (Activation)       (None, 147, 147, 64) 0           batch_normalization_2[0][0]      
__________________________________________________________________________________________________
max_pooling2d_6 (MaxPooling2D)  (None, 73, 73, 64)   0           activation_2[0][0]               
__________________________________________________________________________________________________
conv2d_3 (Conv2D)               (None, 73, 73, 80)   5120        max_pooling2d_6[0][0]            
__________________________________________________________________________________________________
batch_normalization_3 (BatchNor (None, 73, 73, 80)   240         conv2d_3[0][0]                   
__________________________________________________________________________________________________
activation_3 (Activation)       (None, 73, 73, 80)   0           batch_normalization_3[0][0]      
__________________________________________________________________________________________________
conv2d_4 (Conv2D)               (None, 71, 71, 192)  138240      activation_3[0][0]               
__________________________________________________________________________________________________
batch_normalization_4 (BatchNor (None, 71, 71, 192)  576         conv2d_4[0][0]                   
__________________________________________________________________________________________________
activation_4 (Activation)       (None, 71, 71, 192)  0           batch_normalization_4[0][0]      
__________________________________________________________________________________________________
max_pooling2d_7 (MaxPooling2D)  (None, 35, 35, 192)  0           activation_4[0][0]               
__________________________________________________________________________________________________
conv2d_8 (Conv2D)               (None, 35, 35, 64)   12288       max_pooling2d_7[0][0]            
__________________________________________________________________________________________________
batch_normalization_8 (BatchNor (None, 35, 35, 64)   192         conv2d_8[0][0]                   
__________________________________________________________________________________________________
activation_8 (Activation)       (None, 35, 35, 64)   0           batch_normalization_8[0][0]      
__________________________________________________________________________________________________
conv2d_6 (Conv2D)               (None, 35, 35, 48)   9216        max_pooling2d_7[0][0]            
__________________________________________________________________________________________________
conv2d_9 (Conv2D)               (None, 35, 35, 96)   55296       activation_8[0][0]               
__________________________________________________________________________________________________
batch_normalization_6 (BatchNor (None, 35, 35, 48)   144         conv2d_6[0][0]                   
__________________________________________________________________________________________________
batch_normalization_9 (BatchNor (None, 35, 35, 96)   288         conv2d_9[0][0]                   
__________________________________________________________________________________________________
activation_6 (Activation)       (None, 35, 35, 48)   0           batch_normalization_6[0][0]      
__________________________________________________________________________________________________
activation_9 (Activation)       (None, 35, 35, 96)   0           batch_normalization_9[0][0]      
__________________________________________________________________________________________________
average_pooling2d (AveragePooli (None, 35, 35, 192)  0           max_pooling2d_7[0][0]            
__________________________________________________________________________________________________
conv2d_5 (Conv2D)               (None, 35, 35, 64)   12288       max_pooling2d_7[0][0]            
__________________________________________________________________________________________________
conv2d_7 (Conv2D)               (None, 35, 35, 64)   76800       activation_6[0][0]               
__________________________________________________________________________________________________
conv2d_10 (Conv2D)              (None, 35, 35, 96)   82944       activation_9[0][0]               
__________________________________________________________________________________________________
conv2d_11 (Conv2D)              (None, 35, 35, 32)   6144        average_pooling2d[0][0]          
__________________________________________________________________________________________________
batch_normalization_5 (BatchNor (None, 35, 35, 64)   192         conv2d_5[0][0]                   
__________________________________________________________________________________________________
batch_normalization_7 (BatchNor (None, 35, 35, 64)   192         conv2d_7[0][0]                   
__________________________________________________________________________________________________
batch_normalization_10 (BatchNo (None, 35, 35, 96)   288         conv2d_10[0][0]                  
__________________________________________________________________________________________________
batch_normalization_11 (BatchNo (None, 35, 35, 32)   96          conv2d_11[0][0]                  
__________________________________________________________________________________________________
activation_5 (Activation)       (None, 35, 35, 64)   0           batch_normalization_5[0][0]      
__________________________________________________________________________________________________
activation_7 (Activation)       (None, 35, 35, 64)   0           batch_normalization_7[0][0]      
__________________________________________________________________________________________________
activation_10 (Activation)      (None, 35, 35, 96)   0           batch_normalization_10[0][0]     
__________________________________________________________________________________________________
activation_11 (Activation)      (None, 35, 35, 32)   0           batch_normalization_11[0][0]     
__________________________________________________________________________________________________
mixed0 (Concatenate)            (None, 35, 35, 256)  0           activation_5[0][0]               
                                                                 activation_7[0][0]               
                                                                 activation_10[0][0]              
                                                                 activation_11[0][0]              
__________________________________________________________________________________________________
conv2d_15 (Conv2D)              (None, 35, 35, 64)   16384       mixed0[0][0]                     
__________________________________________________________________________________________________
batch_normalization_15 (BatchNo (None, 35, 35, 64)   192         conv2d_15[0][0]                  
__________________________________________________________________________________________________
activation_15 (Activation)      (None, 35, 35, 64)   0           batch_normalization_15[0][0]     
__________________________________________________________________________________________________
conv2d_13 (Conv2D)              (None, 35, 35, 48)   12288       mixed0[0][0]                     
__________________________________________________________________________________________________
conv2d_16 (Conv2D)              (None, 35, 35, 96)   55296       activation_15[0][0]              
__________________________________________________________________________________________________
batch_normalization_13 (BatchNo (None, 35, 35, 48)   144         conv2d_13[0][0]                  
__________________________________________________________________________________________________
batch_normalization_16 (BatchNo (None, 35, 35, 96)   288         conv2d_16[0][0]                  
__________________________________________________________________________________________________
activation_13 (Activation)      (None, 35, 35, 48)   0           batch_normalization_13[0][0]     
__________________________________________________________________________________________________
activation_16 (Activation)      (None, 35, 35, 96)   0           batch_normalization_16[0][0]     
__________________________________________________________________________________________________
average_pooling2d_1 (AveragePoo (None, 35, 35, 256)  0           mixed0[0][0]                     
__________________________________________________________________________________________________
conv2d_12 (Conv2D)              (None, 35, 35, 64)   16384       mixed0[0][0]                     
__________________________________________________________________________________________________
conv2d_14 (Conv2D)              (None, 35, 35, 64)   76800       activation_13[0][0]              
__________________________________________________________________________________________________
conv2d_17 (Conv2D)              (None, 35, 35, 96)   82944       activation_16[0][0]              
__________________________________________________________________________________________________
conv2d_18 (Conv2D)              (None, 35, 35, 64)   16384       average_pooling2d_1[0][0]        
__________________________________________________________________________________________________
batch_normalization_12 (BatchNo (None, 35, 35, 64)   192         conv2d_12[0][0]                  
__________________________________________________________________________________________________
batch_normalization_14 (BatchNo (None, 35, 35, 64)   192         conv2d_14[0][0]                  
__________________________________________________________________________________________________
batch_normalization_17 (BatchNo (None, 35, 35, 96)   288         conv2d_17[0][0]                  
__________________________________________________________________________________________________
batch_normalization_18 (BatchNo (None, 35, 35, 64)   192         conv2d_18[0][0]                  
__________________________________________________________________________________________________
activation_12 (Activation)      (None, 35, 35, 64)   0           batch_normalization_12[0][0]     
__________________________________________________________________________________________________
activation_14 (Activation)      (None, 35, 35, 64)   0           batch_normalization_14[0][0]     
__________________________________________________________________________________________________
activation_17 (Activation)      (None, 35, 35, 96)   0           batch_normalization_17[0][0]     
__________________________________________________________________________________________________
activation_18 (Activation)      (None, 35, 35, 64)   0           batch_normalization_18[0][0]     
__________________________________________________________________________________________________
mixed1 (Concatenate)            (None, 35, 35, 288)  0           activation_12[0][0]              
                                                                 activation_14[0][0]              
                                                                 activation_17[0][0]              
                                                                 activation_18[0][0]              
__________________________________________________________________________________________________
conv2d_22 (Conv2D)              (None, 35, 35, 64)   18432       mixed1[0][0]                     
__________________________________________________________________________________________________
batch_normalization_22 (BatchNo (None, 35, 35, 64)   192         conv2d_22[0][0]                  
__________________________________________________________________________________________________
activation_22 (Activation)      (None, 35, 35, 64)   0           batch_normalization_22[0][0]     
__________________________________________________________________________________________________
conv2d_20 (Conv2D)              (None, 35, 35, 48)   13824       mixed1[0][0]                     
__________________________________________________________________________________________________
conv2d_23 (Conv2D)              (None, 35, 35, 96)   55296       activation_22[0][0]              
__________________________________________________________________________________________________
batch_normalization_20 (BatchNo (None, 35, 35, 48)   144         conv2d_20[0][0]                  
__________________________________________________________________________________________________
batch_normalization_23 (BatchNo (None, 35, 35, 96)   288         conv2d_23[0][0]                  
__________________________________________________________________________________________________
activation_20 (Activation)      (None, 35, 35, 48)   0           batch_normalization_20[0][0]     
__________________________________________________________________________________________________
activation_23 (Activation)      (None, 35, 35, 96)   0           batch_normalization_23[0][0]     
__________________________________________________________________________________________________
average_pooling2d_2 (AveragePoo (None, 35, 35, 288)  0           mixed1[0][0]                     
__________________________________________________________________________________________________
conv2d_19 (Conv2D)              (None, 35, 35, 64)   18432       mixed1[0][0]                     
__________________________________________________________________________________________________
conv2d_21 (Conv2D)              (None, 35, 35, 64)   76800       activation_20[0][0]              
__________________________________________________________________________________________________
conv2d_24 (Conv2D)              (None, 35, 35, 96)   82944       activation_23[0][0]              
__________________________________________________________________________________________________
conv2d_25 (Conv2D)              (None, 35, 35, 64)   18432       average_pooling2d_2[0][0]        
__________________________________________________________________________________________________
batch_normalization_19 (BatchNo (None, 35, 35, 64)   192         conv2d_19[0][0]                  
__________________________________________________________________________________________________
batch_normalization_21 (BatchNo (None, 35, 35, 64)   192         conv2d_21[0][0]                  
__________________________________________________________________________________________________
batch_normalization_24 (BatchNo (None, 35, 35, 96)   288         conv2d_24[0][0]                  
__________________________________________________________________________________________________
batch_normalization_25 (BatchNo (None, 35, 35, 64)   192         conv2d_25[0][0]                  
__________________________________________________________________________________________________
activation_19 (Activation)      (None, 35, 35, 64)   0           batch_normalization_19[0][0]     
__________________________________________________________________________________________________
activation_21 (Activation)      (None, 35, 35, 64)   0           batch_normalization_21[0][0]     
__________________________________________________________________________________________________
activation_24 (Activation)      (None, 35, 35, 96)   0           batch_normalization_24[0][0]     
__________________________________________________________________________________________________
activation_25 (Activation)      (None, 35, 35, 64)   0           batch_normalization_25[0][0]     
__________________________________________________________________________________________________
mixed2 (Concatenate)            (None, 35, 35, 288)  0           activation_19[0][0]              
                                                                 activation_21[0][0]              
                                                                 activation_24[0][0]              
                                                                 activation_25[0][0]              
__________________________________________________________________________________________________
conv2d_27 (Conv2D)              (None, 35, 35, 64)   18432       mixed2[0][0]                     
__________________________________________________________________________________________________
batch_normalization_27 (BatchNo (None, 35, 35, 64)   192         conv2d_27[0][0]                  
__________________________________________________________________________________________________
activation_27 (Activation)      (None, 35, 35, 64)   0           batch_normalization_27[0][0]     
__________________________________________________________________________________________________
conv2d_28 (Conv2D)              (None, 35, 35, 96)   55296       activation_27[0][0]              
__________________________________________________________________________________________________
batch_normalization_28 (BatchNo (None, 35, 35, 96)   288         conv2d_28[0][0]                  
__________________________________________________________________________________________________
activation_28 (Activation)      (None, 35, 35, 96)   0           batch_normalization_28[0][0]     
__________________________________________________________________________________________________
conv2d_26 (Conv2D)              (None, 17, 17, 384)  995328      mixed2[0][0]                     
__________________________________________________________________________________________________
conv2d_29 (Conv2D)              (None, 17, 17, 96)   82944       activation_28[0][0]              
__________________________________________________________________________________________________
batch_normalization_26 (BatchNo (None, 17, 17, 384)  1152        conv2d_26[0][0]                  
__________________________________________________________________________________________________
batch_normalization_29 (BatchNo (None, 17, 17, 96)   288         conv2d_29[0][0]                  
__________________________________________________________________________________________________
activation_26 (Activation)      (None, 17, 17, 384)  0           batch_normalization_26[0][0]     
__________________________________________________________________________________________________
activation_29 (Activation)      (None, 17, 17, 96)   0           batch_normalization_29[0][0]     
__________________________________________________________________________________________________
max_pooling2d_8 (MaxPooling2D)  (None, 17, 17, 288)  0           mixed2[0][0]                     
__________________________________________________________________________________________________
mixed3 (Concatenate)            (None, 17, 17, 768)  0           activation_26[0][0]              
                                                                 activation_29[0][0]              
                                                                 max_pooling2d_8[0][0]            
__________________________________________________________________________________________________
conv2d_34 (Conv2D)              (None, 17, 17, 128)  98304       mixed3[0][0]                     
__________________________________________________________________________________________________
batch_normalization_34 (BatchNo (None, 17, 17, 128)  384         conv2d_34[0][0]                  
__________________________________________________________________________________________________
activation_34 (Activation)      (None, 17, 17, 128)  0           batch_normalization_34[0][0]     
__________________________________________________________________________________________________
conv2d_35 (Conv2D)              (None, 17, 17, 128)  114688      activation_34[0][0]              
__________________________________________________________________________________________________
batch_normalization_35 (BatchNo (None, 17, 17, 128)  384         conv2d_35[0][0]                  
__________________________________________________________________________________________________
activation_35 (Activation)      (None, 17, 17, 128)  0           batch_normalization_35[0][0]     
__________________________________________________________________________________________________
conv2d_31 (Conv2D)              (None, 17, 17, 128)  98304       mixed3[0][0]                     
__________________________________________________________________________________________________
conv2d_36 (Conv2D)              (None, 17, 17, 128)  114688      activation_35[0][0]              
__________________________________________________________________________________________________
batch_normalization_31 (BatchNo (None, 17, 17, 128)  384         conv2d_31[0][0]                  
__________________________________________________________________________________________________
batch_normalization_36 (BatchNo (None, 17, 17, 128)  384         conv2d_36[0][0]                  
__________________________________________________________________________________________________
activation_31 (Activation)      (None, 17, 17, 128)  0           batch_normalization_31[0][0]     
__________________________________________________________________________________________________
activation_36 (Activation)      (None, 17, 17, 128)  0           batch_normalization_36[0][0]     
__________________________________________________________________________________________________
conv2d_32 (Conv2D)              (None, 17, 17, 128)  114688      activation_31[0][0]              
__________________________________________________________________________________________________
conv2d_37 (Conv2D)              (None, 17, 17, 128)  114688      activation_36[0][0]              
__________________________________________________________________________________________________
batch_normalization_32 (BatchNo (None, 17, 17, 128)  384         conv2d_32[0][0]                  
__________________________________________________________________________________________________
batch_normalization_37 (BatchNo (None, 17, 17, 128)  384         conv2d_37[0][0]                  
__________________________________________________________________________________________________
activation_32 (Activation)      (None, 17, 17, 128)  0           batch_normalization_32[0][0]     
__________________________________________________________________________________________________
activation_37 (Activation)      (None, 17, 17, 128)  0           batch_normalization_37[0][0]     
__________________________________________________________________________________________________
average_pooling2d_3 (AveragePoo (None, 17, 17, 768)  0           mixed3[0][0]                     
__________________________________________________________________________________________________
conv2d_30 (Conv2D)              (None, 17, 17, 192)  147456      mixed3[0][0]                     
__________________________________________________________________________________________________
conv2d_33 (Conv2D)              (None, 17, 17, 192)  172032      activation_32[0][0]              
__________________________________________________________________________________________________
conv2d_38 (Conv2D)              (None, 17, 17, 192)  172032      activation_37[0][0]              
__________________________________________________________________________________________________
conv2d_39 (Conv2D)              (None, 17, 17, 192)  147456      average_pooling2d_3[0][0]        
__________________________________________________________________________________________________
batch_normalization_30 (BatchNo (None, 17, 17, 192)  576         conv2d_30[0][0]                  
__________________________________________________________________________________________________
batch_normalization_33 (BatchNo (None, 17, 17, 192)  576         conv2d_33[0][0]                  
__________________________________________________________________________________________________
batch_normalization_38 (BatchNo (None, 17, 17, 192)  576         conv2d_38[0][0]                  
__________________________________________________________________________________________________
batch_normalization_39 (BatchNo (None, 17, 17, 192)  576         conv2d_39[0][0]                  
__________________________________________________________________________________________________
activation_30 (Activation)      (None, 17, 17, 192)  0           batch_normalization_30[0][0]     
__________________________________________________________________________________________________
activation_33 (Activation)      (None, 17, 17, 192)  0           batch_normalization_33[0][0]     
__________________________________________________________________________________________________
activation_38 (Activation)      (None, 17, 17, 192)  0           batch_normalization_38[0][0]     
__________________________________________________________________________________________________
activation_39 (Activation)      (None, 17, 17, 192)  0           batch_normalization_39[0][0]     
__________________________________________________________________________________________________
mixed4 (Concatenate)            (None, 17, 17, 768)  0           activation_30[0][0]              
                                                                 activation_33[0][0]              
                                                                 activation_38[0][0]              
                                                                 activation_39[0][0]              
__________________________________________________________________________________________________
conv2d_44 (Conv2D)              (None, 17, 17, 160)  122880      mixed4[0][0]                     
__________________________________________________________________________________________________
batch_normalization_44 (BatchNo (None, 17, 17, 160)  480         conv2d_44[0][0]                  
__________________________________________________________________________________________________
activation_44 (Activation)      (None, 17, 17, 160)  0           batch_normalization_44[0][0]     
__________________________________________________________________________________________________
conv2d_45 (Conv2D)              (None, 17, 17, 160)  179200      activation_44[0][0]              
__________________________________________________________________________________________________
batch_normalization_45 (BatchNo (None, 17, 17, 160)  480         conv2d_45[0][0]                  
__________________________________________________________________________________________________
activation_45 (Activation)      (None, 17, 17, 160)  0           batch_normalization_45[0][0]     
__________________________________________________________________________________________________
conv2d_41 (Conv2D)              (None, 17, 17, 160)  122880      mixed4[0][0]                     
__________________________________________________________________________________________________
conv2d_46 (Conv2D)              (None, 17, 17, 160)  179200      activation_45[0][0]              
__________________________________________________________________________________________________
batch_normalization_41 (BatchNo (None, 17, 17, 160)  480         conv2d_41[0][0]                  
__________________________________________________________________________________________________
batch_normalization_46 (BatchNo (None, 17, 17, 160)  480         conv2d_46[0][0]                  
__________________________________________________________________________________________________
activation_41 (Activation)      (None, 17, 17, 160)  0           batch_normalization_41[0][0]     
__________________________________________________________________________________________________
activation_46 (Activation)      (None, 17, 17, 160)  0           batch_normalization_46[0][0]     
__________________________________________________________________________________________________
conv2d_42 (Conv2D)              (None, 17, 17, 160)  179200      activation_41[0][0]              
__________________________________________________________________________________________________
conv2d_47 (Conv2D)              (None, 17, 17, 160)  179200      activation_46[0][0]              
__________________________________________________________________________________________________
batch_normalization_42 (BatchNo (None, 17, 17, 160)  480         conv2d_42[0][0]                  
__________________________________________________________________________________________________
batch_normalization_47 (BatchNo (None, 17, 17, 160)  480         conv2d_47[0][0]                  
__________________________________________________________________________________________________
activation_42 (Activation)      (None, 17, 17, 160)  0           batch_normalization_42[0][0]     
__________________________________________________________________________________________________
activation_47 (Activation)      (None, 17, 17, 160)  0           batch_normalization_47[0][0]     
__________________________________________________________________________________________________
average_pooling2d_4 (AveragePoo (None, 17, 17, 768)  0           mixed4[0][0]                     
__________________________________________________________________________________________________
conv2d_40 (Conv2D)              (None, 17, 17, 192)  147456      mixed4[0][0]                     
__________________________________________________________________________________________________
conv2d_43 (Conv2D)              (None, 17, 17, 192)  215040      activation_42[0][0]              
__________________________________________________________________________________________________
conv2d_48 (Conv2D)              (None, 17, 17, 192)  215040      activation_47[0][0]              
__________________________________________________________________________________________________
conv2d_49 (Conv2D)              (None, 17, 17, 192)  147456      average_pooling2d_4[0][0]        
__________________________________________________________________________________________________
batch_normalization_40 (BatchNo (None, 17, 17, 192)  576         conv2d_40[0][0]                  
__________________________________________________________________________________________________
batch_normalization_43 (BatchNo (None, 17, 17, 192)  576         conv2d_43[0][0]                  
__________________________________________________________________________________________________
batch_normalization_48 (BatchNo (None, 17, 17, 192)  576         conv2d_48[0][0]                  
__________________________________________________________________________________________________
batch_normalization_49 (BatchNo (None, 17, 17, 192)  576         conv2d_49[0][0]                  
__________________________________________________________________________________________________
activation_40 (Activation)      (None, 17, 17, 192)  0           batch_normalization_40[0][0]     
__________________________________________________________________________________________________
activation_43 (Activation)      (None, 17, 17, 192)  0           batch_normalization_43[0][0]     
__________________________________________________________________________________________________
activation_48 (Activation)      (None, 17, 17, 192)  0           batch_normalization_48[0][0]     
__________________________________________________________________________________________________
activation_49 (Activation)      (None, 17, 17, 192)  0           batch_normalization_49[0][0]     
__________________________________________________________________________________________________
mixed5 (Concatenate)            (None, 17, 17, 768)  0           activation_40[0][0]              
                                                                 activation_43[0][0]              
                                                                 activation_48[0][0]              
                                                                 activation_49[0][0]              
__________________________________________________________________________________________________
conv2d_54 (Conv2D)              (None, 17, 17, 160)  122880      mixed5[0][0]                     
__________________________________________________________________________________________________
batch_normalization_54 (BatchNo (None, 17, 17, 160)  480         conv2d_54[0][0]                  
__________________________________________________________________________________________________
activation_54 (Activation)      (None, 17, 17, 160)  0           batch_normalization_54[0][0]     
__________________________________________________________________________________________________
conv2d_55 (Conv2D)              (None, 17, 17, 160)  179200      activation_54[0][0]              
__________________________________________________________________________________________________
batch_normalization_55 (BatchNo (None, 17, 17, 160)  480         conv2d_55[0][0]                  
__________________________________________________________________________________________________
activation_55 (Activation)      (None, 17, 17, 160)  0           batch_normalization_55[0][0]     
__________________________________________________________________________________________________
conv2d_51 (Conv2D)              (None, 17, 17, 160)  122880      mixed5[0][0]                     
__________________________________________________________________________________________________
conv2d_56 (Conv2D)              (None, 17, 17, 160)  179200      activation_55[0][0]              
__________________________________________________________________________________________________
batch_normalization_51 (BatchNo (None, 17, 17, 160)  480         conv2d_51[0][0]                  
__________________________________________________________________________________________________
batch_normalization_56 (BatchNo (None, 17, 17, 160)  480         conv2d_56[0][0]                  
__________________________________________________________________________________________________
activation_51 (Activation)      (None, 17, 17, 160)  0           batch_normalization_51[0][0]     
__________________________________________________________________________________________________
activation_56 (Activation)      (None, 17, 17, 160)  0           batch_normalization_56[0][0]     
__________________________________________________________________________________________________
conv2d_52 (Conv2D)              (None, 17, 17, 160)  179200      activation_51[0][0]              
__________________________________________________________________________________________________
conv2d_57 (Conv2D)              (None, 17, 17, 160)  179200      activation_56[0][0]              
__________________________________________________________________________________________________
batch_normalization_52 (BatchNo (None, 17, 17, 160)  480         conv2d_52[0][0]                  
__________________________________________________________________________________________________
batch_normalization_57 (BatchNo (None, 17, 17, 160)  480         conv2d_57[0][0]                  
__________________________________________________________________________________________________
activation_52 (Activation)      (None, 17, 17, 160)  0           batch_normalization_52[0][0]     
__________________________________________________________________________________________________
activation_57 (Activation)      (None, 17, 17, 160)  0           batch_normalization_57[0][0]     
__________________________________________________________________________________________________
average_pooling2d_5 (AveragePoo (None, 17, 17, 768)  0           mixed5[0][0]                     
__________________________________________________________________________________________________
conv2d_50 (Conv2D)              (None, 17, 17, 192)  147456      mixed5[0][0]                     
__________________________________________________________________________________________________
conv2d_53 (Conv2D)              (None, 17, 17, 192)  215040      activation_52[0][0]              
__________________________________________________________________________________________________
conv2d_58 (Conv2D)              (None, 17, 17, 192)  215040      activation_57[0][0]              
__________________________________________________________________________________________________
conv2d_59 (Conv2D)              (None, 17, 17, 192)  147456      average_pooling2d_5[0][0]        
__________________________________________________________________________________________________
batch_normalization_50 (BatchNo (None, 17, 17, 192)  576         conv2d_50[0][0]                  
__________________________________________________________________________________________________
batch_normalization_53 (BatchNo (None, 17, 17, 192)  576         conv2d_53[0][0]                  
__________________________________________________________________________________________________
batch_normalization_58 (BatchNo (None, 17, 17, 192)  576         conv2d_58[0][0]                  
__________________________________________________________________________________________________
batch_normalization_59 (BatchNo (None, 17, 17, 192)  576         conv2d_59[0][0]                  
__________________________________________________________________________________________________
activation_50 (Activation)      (None, 17, 17, 192)  0           batch_normalization_50[0][0]     
__________________________________________________________________________________________________
activation_53 (Activation)      (None, 17, 17, 192)  0           batch_normalization_53[0][0]     
__________________________________________________________________________________________________
activation_58 (Activation)      (None, 17, 17, 192)  0           batch_normalization_58[0][0]     
__________________________________________________________________________________________________
activation_59 (Activation)      (None, 17, 17, 192)  0           batch_normalization_59[0][0]     
__________________________________________________________________________________________________
mixed6 (Concatenate)            (None, 17, 17, 768)  0           activation_50[0][0]              
                                                                 activation_53[0][0]              
                                                                 activation_58[0][0]              
                                                                 activation_59[0][0]              
__________________________________________________________________________________________________
conv2d_64 (Conv2D)              (None, 17, 17, 192)  147456      mixed6[0][0]                     
__________________________________________________________________________________________________
batch_normalization_64 (BatchNo (None, 17, 17, 192)  576         conv2d_64[0][0]                  
__________________________________________________________________________________________________
activation_64 (Activation)      (None, 17, 17, 192)  0           batch_normalization_64[0][0]     
__________________________________________________________________________________________________
conv2d_65 (Conv2D)              (None, 17, 17, 192)  258048      activation_64[0][0]              
__________________________________________________________________________________________________
batch_normalization_65 (BatchNo (None, 17, 17, 192)  576         conv2d_65[0][0]                  
__________________________________________________________________________________________________
activation_65 (Activation)      (None, 17, 17, 192)  0           batch_normalization_65[0][0]     
__________________________________________________________________________________________________
conv2d_61 (Conv2D)              (None, 17, 17, 192)  147456      mixed6[0][0]                     
__________________________________________________________________________________________________
conv2d_66 (Conv2D)              (None, 17, 17, 192)  258048      activation_65[0][0]              
__________________________________________________________________________________________________
batch_normalization_61 (BatchNo (None, 17, 17, 192)  576         conv2d_61[0][0]                  
__________________________________________________________________________________________________
batch_normalization_66 (BatchNo (None, 17, 17, 192)  576         conv2d_66[0][0]                  
__________________________________________________________________________________________________
activation_61 (Activation)      (None, 17, 17, 192)  0           batch_normalization_61[0][0]     
__________________________________________________________________________________________________
activation_66 (Activation)      (None, 17, 17, 192)  0           batch_normalization_66[0][0]     
__________________________________________________________________________________________________
conv2d_62 (Conv2D)              (None, 17, 17, 192)  258048      activation_61[0][0]              
__________________________________________________________________________________________________
conv2d_67 (Conv2D)              (None, 17, 17, 192)  258048      activation_66[0][0]              
__________________________________________________________________________________________________
batch_normalization_62 (BatchNo (None, 17, 17, 192)  576         conv2d_62[0][0]                  
__________________________________________________________________________________________________
batch_normalization_67 (BatchNo (None, 17, 17, 192)  576         conv2d_67[0][0]                  
__________________________________________________________________________________________________
activation_62 (Activation)      (None, 17, 17, 192)  0           batch_normalization_62[0][0]     
__________________________________________________________________________________________________
activation_67 (Activation)      (None, 17, 17, 192)  0           batch_normalization_67[0][0]     
__________________________________________________________________________________________________
average_pooling2d_6 (AveragePoo (None, 17, 17, 768)  0           mixed6[0][0]                     
__________________________________________________________________________________________________
conv2d_60 (Conv2D)              (None, 17, 17, 192)  147456      mixed6[0][0]                     
__________________________________________________________________________________________________
conv2d_63 (Conv2D)              (None, 17, 17, 192)  258048      activation_62[0][0]              
__________________________________________________________________________________________________
conv2d_68 (Conv2D)              (None, 17, 17, 192)  258048      activation_67[0][0]              
__________________________________________________________________________________________________
conv2d_69 (Conv2D)              (None, 17, 17, 192)  147456      average_pooling2d_6[0][0]        
__________________________________________________________________________________________________
batch_normalization_60 (BatchNo (None, 17, 17, 192)  576         conv2d_60[0][0]                  
__________________________________________________________________________________________________
batch_normalization_63 (BatchNo (None, 17, 17, 192)  576         conv2d_63[0][0]                  
__________________________________________________________________________________________________
batch_normalization_68 (BatchNo (None, 17, 17, 192)  576         conv2d_68[0][0]                  
__________________________________________________________________________________________________
batch_normalization_69 (BatchNo (None, 17, 17, 192)  576         conv2d_69[0][0]                  
__________________________________________________________________________________________________
activation_60 (Activation)      (None, 17, 17, 192)  0           batch_normalization_60[0][0]     
__________________________________________________________________________________________________
activation_63 (Activation)      (None, 17, 17, 192)  0           batch_normalization_63[0][0]     
__________________________________________________________________________________________________
activation_68 (Activation)      (None, 17, 17, 192)  0           batch_normalization_68[0][0]     
__________________________________________________________________________________________________
activation_69 (Activation)      (None, 17, 17, 192)  0           batch_normalization_69[0][0]     
__________________________________________________________________________________________________
mixed7 (Concatenate)            (None, 17, 17, 768)  0           activation_60[0][0]              
                                                                 activation_63[0][0]              
                                                                 activation_68[0][0]              
                                                                 activation_69[0][0]              
__________________________________________________________________________________________________
conv2d_72 (Conv2D)              (None, 17, 17, 192)  147456      mixed7[0][0]                     
__________________________________________________________________________________________________
batch_normalization_72 (BatchNo (None, 17, 17, 192)  576         conv2d_72[0][0]                  
__________________________________________________________________________________________________
activation_72 (Activation)      (None, 17, 17, 192)  0           batch_normalization_72[0][0]     
__________________________________________________________________________________________________
conv2d_73 (Conv2D)              (None, 17, 17, 192)  258048      activation_72[0][0]              
__________________________________________________________________________________________________
batch_normalization_73 (BatchNo (None, 17, 17, 192)  576         conv2d_73[0][0]                  
__________________________________________________________________________________________________
activation_73 (Activation)      (None, 17, 17, 192)  0           batch_normalization_73[0][0]     
__________________________________________________________________________________________________
conv2d_70 (Conv2D)              (None, 17, 17, 192)  147456      mixed7[0][0]                     
__________________________________________________________________________________________________
conv2d_74 (Conv2D)              (None, 17, 17, 192)  258048      activation_73[0][0]              
__________________________________________________________________________________________________
batch_normalization_70 (BatchNo (None, 17, 17, 192)  576         conv2d_70[0][0]                  
__________________________________________________________________________________________________
batch_normalization_74 (BatchNo (None, 17, 17, 192)  576         conv2d_74[0][0]                  
__________________________________________________________________________________________________
activation_70 (Activation)      (None, 17, 17, 192)  0           batch_normalization_70[0][0]     
__________________________________________________________________________________________________
activation_74 (Activation)      (None, 17, 17, 192)  0           batch_normalization_74[0][0]     
__________________________________________________________________________________________________
conv2d_71 (Conv2D)              (None, 8, 8, 320)    552960      activation_70[0][0]              
__________________________________________________________________________________________________
conv2d_75 (Conv2D)              (None, 8, 8, 192)    331776      activation_74[0][0]              
__________________________________________________________________________________________________
batch_normalization_71 (BatchNo (None, 8, 8, 320)    960         conv2d_71[0][0]                  
__________________________________________________________________________________________________
batch_normalization_75 (BatchNo (None, 8, 8, 192)    576         conv2d_75[0][0]                  
__________________________________________________________________________________________________
activation_71 (Activation)      (None, 8, 8, 320)    0           batch_normalization_71[0][0]     
__________________________________________________________________________________________________
activation_75 (Activation)      (None, 8, 8, 192)    0           batch_normalization_75[0][0]     
__________________________________________________________________________________________________
max_pooling2d_9 (MaxPooling2D)  (None, 8, 8, 768)    0           mixed7[0][0]                     
__________________________________________________________________________________________________
mixed8 (Concatenate)            (None, 8, 8, 1280)   0           activation_71[0][0]              
                                                                 activation_75[0][0]              
                                                                 max_pooling2d_9[0][0]            
__________________________________________________________________________________________________
conv2d_80 (Conv2D)              (None, 8, 8, 448)    573440      mixed8[0][0]                     
__________________________________________________________________________________________________
batch_normalization_80 (BatchNo (None, 8, 8, 448)    1344        conv2d_80[0][0]                  
__________________________________________________________________________________________________
activation_80 (Activation)      (None, 8, 8, 448)    0           batch_normalization_80[0][0]     
__________________________________________________________________________________________________
conv2d_77 (Conv2D)              (None, 8, 8, 384)    491520      mixed8[0][0]                     
__________________________________________________________________________________________________
conv2d_81 (Conv2D)              (None, 8, 8, 384)    1548288     activation_80[0][0]              
__________________________________________________________________________________________________
batch_normalization_77 (BatchNo (None, 8, 8, 384)    1152        conv2d_77[0][0]                  
__________________________________________________________________________________________________
batch_normalization_81 (BatchNo (None, 8, 8, 384)    1152        conv2d_81[0][0]                  
__________________________________________________________________________________________________
activation_77 (Activation)      (None, 8, 8, 384)    0           batch_normalization_77[0][0]     
__________________________________________________________________________________________________
activation_81 (Activation)      (None, 8, 8, 384)    0           batch_normalization_81[0][0]     
__________________________________________________________________________________________________
conv2d_78 (Conv2D)              (None, 8, 8, 384)    442368      activation_77[0][0]              
__________________________________________________________________________________________________
conv2d_79 (Conv2D)              (None, 8, 8, 384)    442368      activation_77[0][0]              
__________________________________________________________________________________________________
conv2d_82 (Conv2D)              (None, 8, 8, 384)    442368      activation_81[0][0]              
__________________________________________________________________________________________________
conv2d_83 (Conv2D)              (None, 8, 8, 384)    442368      activation_81[0][0]              
__________________________________________________________________________________________________
average_pooling2d_7 (AveragePoo (None, 8, 8, 1280)   0           mixed8[0][0]                     
__________________________________________________________________________________________________
conv2d_76 (Conv2D)              (None, 8, 8, 320)    409600      mixed8[0][0]                     
__________________________________________________________________________________________________
batch_normalization_78 (BatchNo (None, 8, 8, 384)    1152        conv2d_78[0][0]                  
__________________________________________________________________________________________________
batch_normalization_79 (BatchNo (None, 8, 8, 384)    1152        conv2d_79[0][0]                  
__________________________________________________________________________________________________
batch_normalization_82 (BatchNo (None, 8, 8, 384)    1152        conv2d_82[0][0]                  
__________________________________________________________________________________________________
batch_normalization_83 (BatchNo (None, 8, 8, 384)    1152        conv2d_83[0][0]                  
__________________________________________________________________________________________________
conv2d_84 (Conv2D)              (None, 8, 8, 192)    245760      average_pooling2d_7[0][0]        
__________________________________________________________________________________________________
batch_normalization_76 (BatchNo (None, 8, 8, 320)    960         conv2d_76[0][0]                  
__________________________________________________________________________________________________
activation_78 (Activation)      (None, 8, 8, 384)    0           batch_normalization_78[0][0]     
__________________________________________________________________________________________________
activation_79 (Activation)      (None, 8, 8, 384)    0           batch_normalization_79[0][0]     
__________________________________________________________________________________________________
activation_82 (Activation)      (None, 8, 8, 384)    0           batch_normalization_82[0][0]     
__________________________________________________________________________________________________
activation_83 (Activation)      (None, 8, 8, 384)    0           batch_normalization_83[0][0]     
__________________________________________________________________________________________________
batch_normalization_84 (BatchNo (None, 8, 8, 192)    576         conv2d_84[0][0]                  
__________________________________________________________________________________________________
activation_76 (Activation)      (None, 8, 8, 320)    0           batch_normalization_76[0][0]     
__________________________________________________________________________________________________
mixed9_0 (Concatenate)          (None, 8, 8, 768)    0           activation_78[0][0]              
                                                                 activation_79[0][0]              
__________________________________________________________________________________________________
concatenate (Concatenate)       (None, 8, 8, 768)    0           activation_82[0][0]              
                                                                 activation_83[0][0]              
__________________________________________________________________________________________________
activation_84 (Activation)      (None, 8, 8, 192)    0           batch_normalization_84[0][0]     
__________________________________________________________________________________________________
mixed9 (Concatenate)            (None, 8, 8, 2048)   0           activation_76[0][0]              
                                                                 mixed9_0[0][0]                   
                                                                 concatenate[0][0]                
                                                                 activation_84[0][0]              
__________________________________________________________________________________________________
conv2d_89 (Conv2D)              (None, 8, 8, 448)    917504      mixed9[0][0]                     
__________________________________________________________________________________________________
batch_normalization_89 (BatchNo (None, 8, 8, 448)    1344        conv2d_89[0][0]                  
__________________________________________________________________________________________________
activation_89 (Activation)      (None, 8, 8, 448)    0           batch_normalization_89[0][0]     
__________________________________________________________________________________________________
conv2d_86 (Conv2D)              (None, 8, 8, 384)    786432      mixed9[0][0]                     
__________________________________________________________________________________________________
conv2d_90 (Conv2D)              (None, 8, 8, 384)    1548288     activation_89[0][0]              
__________________________________________________________________________________________________
batch_normalization_86 (BatchNo (None, 8, 8, 384)    1152        conv2d_86[0][0]                  
__________________________________________________________________________________________________
batch_normalization_90 (BatchNo (None, 8, 8, 384)    1152        conv2d_90[0][0]                  
__________________________________________________________________________________________________
activation_86 (Activation)      (None, 8, 8, 384)    0           batch_normalization_86[0][0]     
__________________________________________________________________________________________________
activation_90 (Activation)      (None, 8, 8, 384)    0           batch_normalization_90[0][0]     
__________________________________________________________________________________________________
conv2d_87 (Conv2D)              (None, 8, 8, 384)    442368      activation_86[0][0]              
__________________________________________________________________________________________________
conv2d_88 (Conv2D)              (None, 8, 8, 384)    442368      activation_86[0][0]              
__________________________________________________________________________________________________
conv2d_91 (Conv2D)              (None, 8, 8, 384)    442368      activation_90[0][0]              
__________________________________________________________________________________________________
conv2d_92 (Conv2D)              (None, 8, 8, 384)    442368      activation_90[0][0]              
__________________________________________________________________________________________________
average_pooling2d_8 (AveragePoo (None, 8, 8, 2048)   0           mixed9[0][0]                     
__________________________________________________________________________________________________
conv2d_85 (Conv2D)              (None, 8, 8, 320)    655360      mixed9[0][0]                     
__________________________________________________________________________________________________
batch_normalization_87 (BatchNo (None, 8, 8, 384)    1152        conv2d_87[0][0]                  
__________________________________________________________________________________________________
batch_normalization_88 (BatchNo (None, 8, 8, 384)    1152        conv2d_88[0][0]                  
__________________________________________________________________________________________________
batch_normalization_91 (BatchNo (None, 8, 8, 384)    1152        conv2d_91[0][0]                  
__________________________________________________________________________________________________
batch_normalization_92 (BatchNo (None, 8, 8, 384)    1152        conv2d_92[0][0]                  
__________________________________________________________________________________________________
conv2d_93 (Conv2D)              (None, 8, 8, 192)    393216      average_pooling2d_8[0][0]        
__________________________________________________________________________________________________
batch_normalization_85 (BatchNo (None, 8, 8, 320)    960         conv2d_85[0][0]                  
__________________________________________________________________________________________________
activation_87 (Activation)      (None, 8, 8, 384)    0           batch_normalization_87[0][0]     
__________________________________________________________________________________________________
activation_88 (Activation)      (None, 8, 8, 384)    0           batch_normalization_88[0][0]     
__________________________________________________________________________________________________
activation_91 (Activation)      (None, 8, 8, 384)    0           batch_normalization_91[0][0]     
__________________________________________________________________________________________________
activation_92 (Activation)      (None, 8, 8, 384)    0           batch_normalization_92[0][0]     
__________________________________________________________________________________________________
batch_normalization_93 (BatchNo (None, 8, 8, 192)    576         conv2d_93[0][0]                  
__________________________________________________________________________________________________
activation_85 (Activation)      (None, 8, 8, 320)    0           batch_normalization_85[0][0]     
__________________________________________________________________________________________________
mixed9_1 (Concatenate)          (None, 8, 8, 768)    0           activation_87[0][0]              
                                                                 activation_88[0][0]              
__________________________________________________________________________________________________
concatenate_1 (Concatenate)     (None, 8, 8, 768)    0           activation_91[0][0]              
                                                                 activation_92[0][0]              
__________________________________________________________________________________________________
activation_93 (Activation)      (None, 8, 8, 192)    0           batch_normalization_93[0][0]     
__________________________________________________________________________________________________
mixed10 (Concatenate)           (None, 8, 8, 2048)   0           activation_85[0][0]              
                                                                 mixed9_1[0][0]                   
                                                                 concatenate_1[0][0]              
                                                                 activation_93[0][0]              
__________________________________________________________________________________________________
avg_pool (GlobalAveragePooling2 (None, 2048)         0           mixed10[0][0]                    
__________________________________________________________________________________________________
predictions (Dense)             (None, 1000)         2049000     avg_pool[0][0]                   
==================================================================================================
Total params: 23,851,784
Trainable params: 23,817,352
Non-trainable params: 34,432
__________________________________________________________________________________________________

concatenate는 원본 값을 그대로 유지하고 싶을 때 (구조를 합침)
add는 합쳐서 하나의 결과를 유지한다

원본 이미지와 조건을 더할때 일반적으로 concatenate를 사용하는 것이 유용한다
add를 사용해야 할때는 concatenate를 하고나서도 해도 되기 때문이다

import numpy as np

a = np.array([[1,2,3]])
b = np.array([[1,2,3]])

a+b
array([[2, 4, 6]])

c = np.concatenate((a,b))

c
# array([[1, 2, 3],
#        [1, 2, 3]])

c.sum(axis=0) 
# array([2, 4, 6])

cGAN의 목표

이러한 과정을 통해 G는 입력 이미지 x와 noise z로 부터 출력 이미지 y로 매핑하도록 학습이 된다

Pix2Pix 구현

import tensorflow as tf

import os
import pathlib
import time
import datetime

from matplotlib import pyplot as plt
from IPython import display

dataset_name = "facades"

_URL = f'http://efrosgans.eecs.berkeley.edu/pix2pix/datasets/{dataset_name}.tar.gz'

path_to_zip = tf.keras.utils.get_file(
    fname=f"{dataset_name}.tar.gz",
    origin=_URL,
    extract=True)

path_to_zip  = pathlib.Path(path_to_zip)

PATH = path_to_zip.parent/dataset_name

list(PATH.parent.iterdir())

[PosixPath('/Users/jihyeokjeong/.keras/datasets/imdb_word_index.json'),
 PosixPath('/Users/jihyeokjeong/.keras/datasets/mnist.npz'),
 PosixPath('/Users/jihyeokjeong/.keras/datasets/fashion-mnist'),
 PosixPath('/Users/jihyeokjeong/.keras/datasets/imdb.npz'),
 PosixPath('/Users/jihyeokjeong/.keras/datasets/facades'),
 PosixPath('/Users/jihyeokjeong/.keras/datasets/facades.tar.gz')]

sample_image = tf.io.read_file(str(PATH / 'train/1.jpg'))
sample_image = tf.io.decode_jpeg(sample_image)
print(sample_image.shape)
# (256, 512, 3)

plt.figure()
plt.imshow(sample_image)

def load(image_file):
  image = tf.io.read_file(image_file)
  image = tf.image.decode_jpeg(image)
    
  w = tf.shape(image)[1]
  w = w // 2
  input_image = image[:, w:, :]
  real_image = image[:, :w, :]

  input_image = tf.cast(input_image, tf.float32)
  real_image = tf.cast(real_image, tf.float32)

  return input_image, real_image

inp, re = load(str(PATH / 'train/100.jpg'))

plt.figure()
plt.imshow(inp / 255.0)
plt.figure()
plt.imshow(re / 255.0)

BUFFER_SIZE = 400
BATCH_SIZE = 1
IMG_WIDTH = 256
IMG_HEIGHT = 256

def resize(input_image, real_image, height, width):
  input_image = tf.image.resize(input_image, [height, width],
                                method=tf.image.ResizeMethod.NEAREST_NEIGHBOR)
  real_image = tf.image.resize(real_image, [height, width],
                               method=tf.image.ResizeMethod.NEAREST_NEIGHBOR)

  return input_image, real_image

def random_crop(input_image, real_image):
  stacked_image = tf.stack([input_image, real_image], axis=0)
  cropped_image = tf.image.random_crop(
      stacked_image, size=[2, IMG_HEIGHT, IMG_WIDTH, 3])

  return cropped_image[0], cropped_image[1]

def normalize(input_image, real_image):
  input_image = (input_image / 127.5) - 1
  real_image = (real_image / 127.5) - 1

  return input_image, real_image

@tf.function()
def random_jitter(input_image, real_image):
  input_image, real_image = resize(input_image, real_image, 286, 286)

  input_image, real_image = random_crop(input_image, real_image)

  if tf.random.uniform(()) > 0.5:
    input_image = tf.image.flip_left_right(input_image)
    real_image = tf.image.flip_left_right(real_image)

  return input_image, real_image

plt.figure(figsize=(6, 6))
for i in range(4):
  rj_inp, rj_re = random_jitter(inp, re)
  plt.subplot(2, 2, i + 1)
  plt.imshow(rj_inp / 255.0)
  plt.axis('off')
plt.show()

def load_image_train(image_file):
  input_image, real_image = load(image_file)
  input_image, real_image = random_jitter(input_image, real_image)
  input_image, real_image = normalize(input_image, real_image)

  return input_image, real_image

def load_image_test(image_file):
  input_image, real_image = load(image_file)
  input_image, real_image = resize(input_image, real_image,
                                   IMG_HEIGHT, IMG_WIDTH)
  input_image, real_image = normalize(input_image, real_image)

  return input_image, real_image

train_dataset = tf.data.Dataset.list_files(str(PATH / 'train/*.jpg'))
train_dataset = train_dataset.map(load_image_train,
                                  num_parallel_calls=tf.data.AUTOTUNE)
train_dataset = train_dataset.shuffle(BUFFER_SIZE)
train_dataset = train_dataset.batch(BATCH_SIZE)

try:
  test_dataset = tf.data.Dataset.list_files(str(PATH / 'test/*.jpg'))
except tf.errors.InvalidArgumentError:
  test_dataset = tf.data.Dataset.list_files(str(PATH / 'val/*.jpg'))
test_dataset = test_dataset.map(load_image_test)
test_dataset = test_dataset.batch(BATCH_SIZE)

OUTPUT_CHANNELS = 3

def downsample(filters, size, apply_batchnorm=True):
  initializer = tf.random_normal_initializer(0., 0.02)

  result = tf.keras.Sequential()
  result.add(
      tf.keras.layers.Conv2D(filters, size, strides=2, padding='same',
                             kernel_initializer=initializer, use_bias=False)) # bias 사용 옵션

  if apply_batchnorm:
    result.add(tf.keras.layers.BatchNormalization()) # BM는 옵션

  result.add(tf.keras.layers.LeakyReLU())

  return result

down_model = downsample(3, 4)
down_result = down_model(tf.expand_dims(inp, 0))
print (down_result.shape)
# (1, 128, 128, 3)

def upsample(filters, size, apply_dropout=False):
  initializer = tf.random_normal_initializer(0., 0.02)

  result = tf.keras.Sequential()
  result.add(
    tf.keras.layers.Conv2DTranspose(filters, size, strides=2,
                                    padding='same',
                                    kernel_initializer=initializer,
                                    use_bias=False)) 
  # bias를 사용하지 않는 이유 : GAN은 기본적으로 학습이 잘 안되기 때문에 간소화 시켜야 하므로 bias를 사용하지 않는다
  
  result.add(tf.keras.layers.BatchNormalization())
  
  # layer가 많을 때는 dropout을 사용하지 않아도 되기 때문에 옵션으로 사용 
  # 범용적 사용을 위해 옵션으로 만듦
  if apply_dropout:
      result.add(tf.keras.layers.Dropout(0.5))

  result.add(tf.keras.layers.ReLU())
  
  return result

up_model = upsample(3, 4)
up_result = up_model(down_result)
print (up_result.shape)
# (1, 256, 256, 3)

# Generator안에서 U-net 생성
# U-net을 가짜 데이터 생성용으로 사용
def Generator():
  inputs = tf.keras.layers.Input(shape=[256, 256, 3])

  down_stack = [
    # U-net구조를 위해 size확인을 해야 한다
    downsample(64, 4, apply_batchnorm=False),  # (batch_size, 128, 128, 64)
    downsample(128, 4),  # (batch_size, 64, 64, 128)
    downsample(256, 4),  # (batch_size, 32, 32, 256)
    downsample(512, 4),  # (batch_size, 16, 16, 512)
    downsample(512, 4),  # (batch_size, 8, 8, 512)
    downsample(512, 4),  # (batch_size, 4, 4, 512)
    downsample(512, 4),  # (batch_size, 2, 2, 512)
    downsample(512, 4),  # (batch_size, 1, 1, 512)
  ]

  up_stack = [
    upsample(512, 4, apply_dropout=True),  # (batch_size, 2, 2, 1024)
    upsample(512, 4, apply_dropout=True),  # (batch_size, 4, 4, 1024)
    upsample(512, 4, apply_dropout=True),  # (batch_size, 8, 8, 1024)
    upsample(512, 4),  # (batch_size, 16, 16, 1024)
    upsample(256, 4),  # (batch_size, 32, 32, 512)
    upsample(128, 4),  # (batch_size, 64, 64, 256)
    upsample(64, 4),  # (batch_size, 128, 128, 128)
  ]

  initializer = tf.random_normal_initializer(0., 0.02)
  last = tf.keras.layers.Conv2DTranspose(OUTPUT_CHANNELS, 4,
                                         strides=2,
                                         padding='same',
                                         kernel_initializer=initializer,
                                         activation='tanh')  # (batch_size, 256, 256, 3)
                            # sigmoid를 사용하면 zero centered가 되지 않아 학습이 잘 안되기 때문에 tanh를 사용한다
                            # zero centered가 되지 않으면 약간의 변화에도 값이 크게 변하기 때문에 학습이 잘 안될 수 있다 
  x = inputs

  skips = []
  for down in down_stack:
    x = down(x)
    skips.append(x)

  skips = reversed(skips[:-1])

  for up, skip in zip(up_stack, skips):
    x = up(x)
    x = tf.keras.layers.Concatenate()([x, skip])

  x = last(x)

  return tf.keras.Model(inputs=inputs, outputs=x)

AutoEncoder, U-net 구조는 loss를 어떻게 하느냐에 따라, 학습 데이터를 어떻게 하느냐에 따라 목적이 달라질 수 있다
U-net구조를 통해 segmentation을 사용하기도 하고, 원본 이미지를 복원하기도 하고, 원본 이미지의 색상을 바꾸기도 한다

generator = Generator()

gen_output = generator(inp[tf.newaxis, ...], training=False)
plt.imshow(gen_output[0, ...])

LAMBDA = 100
# tanh을 사용했기 때문에 BinaryCrossentropy을 사용
loss_object = tf.keras.losses.BinaryCrossentropy(from_logits=True)

# One-sided label smoothing: 실제 데이터에 대한 target 값을 1보다 약간 작은 값으로 설정하는 방식 
# 여기서는 one-sided label smoothing 방식을 사용하지 않았다
def generator_loss(disc_generated_output, gen_output, target):
  gan_loss = loss_object(tf.ones_like(disc_generated_output), disc_generated_output)

  l1_loss = tf.reduce_mean(tf.abs(target - gen_output))

  total_gen_loss = gan_loss + (LAMBDA * l1_loss)

  return total_gen_loss, gan_loss, l1_loss

def Discriminator():
  initializer = tf.random_normal_initializer(0., 0.02)

  inp = tf.keras.layers.Input(shape=[256, 256, 3], name='input_image')
  tar = tf.keras.layers.Input(shape=[256, 256, 3], name='target_image')
  
  # 실제값 + 목표 이미지 concatenate
  x = tf.keras.layers.concatenate([inp, tar])  # (batch_size, 256, 256, channels*2)

  down1 = downsample(64, 4, False)(x)  # (batch_size, 128, 128, 64)
  down2 = downsample(128, 4)(down1)  # (batch_size, 64, 64, 128)
  down3 = downsample(256, 4)(down2)  # (batch_size, 32, 32, 256)

  # 출력의 각 30x30 크기 이미지 패치는 입력 이미지의 70x70부분을 판별한다 (부분으로 쪼개어 판별한다/사이즈는 실험을 통해 알아내야 한다)
  # 크기 맞추는 방법을 사용한다 
  zero_pad1 = tf.keras.layers.ZeroPadding2D()(down3)  # (batch_size, 34, 34, 256)
  conv = tf.keras.layers.Conv2D(512, 4, strides=1,
                                kernel_initializer=initializer,
                                use_bias=False)(zero_pad1)  # (batch_size, 31, 31, 512)

  batchnorm1 = tf.keras.layers.BatchNormalization()(conv)

  leaky_relu = tf.keras.layers.LeakyReLU()(batchnorm1)

  zero_pad2 = tf.keras.layers.ZeroPadding2D()(leaky_relu)  # (batch_size, 33, 33, 512)

  last = tf.keras.layers.Conv2D(1, 4, strides=1,
                                kernel_initializer=initializer)(zero_pad2)  # (batch_size, 30, 30, 1)

  return tf.keras.Model(inputs=[inp, tar], outputs=last)

discriminator = Discriminator()

disc_out = discriminator([inp[tf.newaxis, ...], gen_output], training=False)
plt.imshow(disc_out[0, ..., -1], vmin=-20, vmax=20, cmap='RdBu_r')
plt.colorbar()

def discriminator_loss(disc_real_output, disc_generated_output):
  # 진짜를 진짜로 1
  real_loss = loss_object(tf.ones_like(disc_real_output), disc_real_output)
  
  # 가짜는 가짜로 0
  generated_loss = loss_object(tf.zeros_like(disc_generated_output), disc_generated_output)

  total_disc_loss = real_loss + generated_loss

  return total_disc_loss

generator_optimizer = tf.keras.optimizers.Adam(2e-4, beta_1=0.5)
discriminator_optimizer = tf.keras.optimizers.Adam(2e-4, beta_1=0.5)

checkpoint_dir = './training_checkpoints'
checkpoint_prefix = os.path.join(checkpoint_dir, "ckpt")
checkpoint = tf.train.Checkpoint(generator_optimizer=generator_optimizer,
                                 discriminator_optimizer=discriminator_optimizer,
                                 generator=generator,
                                 discriminator=discriminator)

def generate_images(model, test_input, tar):
  prediction = model(test_input, training=True)
  plt.figure(figsize=(15, 15))

  display_list = [test_input[0], tar[0], prediction[0]]
  title = ['Input Image', 'Ground Truth', 'Predicted Image']

  for i in range(3):
    plt.subplot(1, 3, i+1)
    plt.title(title[i])
    plt.imshow(display_list[i] * 0.5 + 0.5)
    plt.axis('off')
  plt.show()

for example_input, example_target in test_dataset.take(1):
  generate_images(generator, example_input, example_target)

log_dir="logs/"

summary_writer = tf.summary.create_file_writer(
  log_dir + "fit/" + datetime.datetime.now().strftime("%Y%m%d-%H%M%S"))

@tf.function
def train_step(input_image, target, step):
  # generator가 먼저 실행되지만 discriminator가 먼저 전체가 실행된다 (with 중첩 때문에)
  # discriminator 학습 -> generator 학습 -> discriminator 학습 -> generator 학습 반복 
  with tf.GradientTape() as gen_tape, tf.GradientTape() as disc_tape:
    # 미분할 대상을 지정한다
    gen_output = generator(input_image, training=True)

    disc_real_output = discriminator([input_image, target], training=True)
    disc_generated_output = discriminator([input_image, gen_output], training=True)

    gen_total_loss, gen_gan_loss, gen_l1_loss = generator_loss(disc_generated_output, gen_output, target)
    disc_loss = discriminator_loss(disc_real_output, disc_generated_output)

  generator_gradients = gen_tape.gradient(gen_total_loss,
                                          generator.trainable_variables)
  discriminator_gradients = disc_tape.gradient(disc_loss,
                                               discriminator.trainable_variables)

  generator_optimizer.apply_gradients(zip(generator_gradients,
                                          generator.trainable_variables))
  discriminator_optimizer.apply_gradients(zip(discriminator_gradients,
                                              discriminator.trainable_variables))

  with summary_writer.as_default():
    tf.summary.scalar('gen_total_loss', gen_total_loss, step=step//1000)
    tf.summary.scalar('gen_gan_loss', gen_gan_loss, step=step//1000)
    tf.summary.scalar('gen_l1_loss', gen_l1_loss, step=step//1000)
    tf.summary.scalar('disc_loss', disc_loss, step=step//1000)

with A:
    with B:
        X()
    Y()

1. A __enter__
2. B __enter__
3. X()
4. B __exit__
5. Y()
6. A __exit__

def fit(train_ds, test_ds, steps):
  example_input, example_target = next(iter(test_ds.take(1)))
  start = time.time()

  for step, (input_image, target) in train_ds.repeat().take(steps).enumerate():
    if (step) % 1000 == 0:
      display.clear_output(wait=True)

      if step != 0:
        print(f'Time taken for 1000 steps: {time.time()-start:.2f} sec\n')

      start = time.time()

      generate_images(generator, example_input, example_target)
      print(f"Step: {step//1000}k")

    train_step(input_image, target, step)

    if (step+1) % 10 == 0:
      print('.', end='', flush=True)

    if (step + 1) % 5000 == 0:
      checkpoint.save(file_prefix=checkpoint_prefix)

# 총 weight 업데이트를 4만번 (GAN은 기본적으로 학습이 잘 안되기 때문에 Epochs을 높게 잡아야 한다)
fit(train_dataset, test_dataset, steps=40000)
# Time taken for 1000 steps: 159.76 sec

for inp, tar in test_dataset.take(5):
  generate_images(generator, inp, tar)

for inp, tar in test_dataset.take(6):
  generate_images(generator, inp, tar)

저작자표시 비영리

'Computer_Science > Visual Intelligence' 카테고리의 다른 글

38일차 - Style Transfer (0)	2021.11.16
37일차 - 논문 수업 (Segmentation) (0)	2021.11.16
36일차 - 논문 수업 (Segmentation) (0)	2021.11.16
35일차 - 논문 수업 (Segmentation) (0)	2021.11.01
34일차 - 논문 수업 (VAE) (0)	2021.11.01

38일차 - Style Transfer

2021. 11. 16. 19:21

728x90

Transfer learning 정리

1. Feature extraction
- CNN (Convolution 연산 부분만)을 활용하여 추출한다 => window 연산을 하기 때문에 입력 개수가 정해지지 않아도 된다
- input_shape을 정하지 않아도 되지만 tensorflow/keras에서 build하기 위해 고정하는 경우가 있다
- Feature extracion을 사용하는 경우 전체를 쓰는 것보다 성능이 일반적으로 좋지 않다. 하지만 데이터가 적을 때 최소한 학습한 내용이 있기 때문에 조금의 영향력을 가질 수 있다 (fine tuning이 필요하다)
2. Fine tuning
- Learning rate를 크게하면 catastropic forgetting이 발생되기 때문에 learning rate를 작게 하여 재학습 시켜야 한다

U-net 구현하기

import tensorflow as tf 
import matplotlib.pyplot as plt
import numpy as np

base_model = tf.keras.applications.MobileNetV2(include_top=False)

base_model.summary()
Model: "mobilenetv2_1.00_224"
__________________________________________________________________________________________________
Layer (type)                    Output Shape         Param #     Connected to                     
==================================================================================================
input_12 (InputLayer)           [(None, None, None,  0                                            
__________________________________________________________________________________________________
Conv1 (Conv2D)                  (None, None, None, 3 864         input_12[0][0]                   
__________________________________________________________________________________________________
bn_Conv1 (BatchNormalization)   (None, None, None, 3 128         Conv1[0][0]                      
__________________________________________________________________________________________________
Conv1_relu (ReLU)               (None, None, None, 3 0           bn_Conv1[0][0]                   
__________________________________________________________________________________________________
expanded_conv_depthwise (Depthw (None, None, None, 3 288         Conv1_relu[0][0]                 
__________________________________________________________________________________________________
expanded_conv_depthwise_BN (Bat (None, None, None, 3 128         expanded_conv_depthwise[0][0]    
__________________________________________________________________________________________________
expanded_conv_depthwise_relu (R (None, None, None, 3 0           expanded_conv_depthwise_BN[0][0] 
__________________________________________________________________________________________________
expanded_conv_project (Conv2D)  (None, None, None, 1 512         expanded_conv_depthwise_relu[0][0
__________________________________________________________________________________________________
expanded_conv_project_BN (Batch (None, None, None, 1 64          expanded_conv_project[0][0]      
__________________________________________________________________________________________________
block_1_expand (Conv2D)         (None, None, None, 9 1536        expanded_conv_project_BN[0][0]   
__________________________________________________________________________________________________
block_1_expand_BN (BatchNormali (None, None, None, 9 384         block_1_expand[0][0]             
__________________________________________________________________________________________________
block_1_expand_relu (ReLU)      (None, None, None, 9 0           block_1_expand_BN[0][0]          
__________________________________________________________________________________________________
block_1_pad (ZeroPadding2D)     (None, None, None, 9 0           block_1_expand_relu[0][0]        
__________________________________________________________________________________________________
block_1_depthwise (DepthwiseCon (None, None, None, 9 864         block_1_pad[0][0]                
__________________________________________________________________________________________________
block_1_depthwise_BN (BatchNorm (None, None, None, 9 384         block_1_depthwise[0][0]          
__________________________________________________________________________________________________
block_1_depthwise_relu (ReLU)   (None, None, None, 9 0           block_1_depthwise_BN[0][0]       
__________________________________________________________________________________________________
block_1_project (Conv2D)        (None, None, None, 2 2304        block_1_depthwise_relu[0][0]     
__________________________________________________________________________________________________
block_1_project_BN (BatchNormal (None, None, None, 2 96          block_1_project[0][0]            
__________________________________________________________________________________________________
block_2_expand (Conv2D)         (None, None, None, 1 3456        block_1_project_BN[0][0]         
__________________________________________________________________________________________________
block_2_expand_BN (BatchNormali (None, None, None, 1 576         block_2_expand[0][0]             
__________________________________________________________________________________________________
block_2_expand_relu (ReLU)      (None, None, None, 1 0           block_2_expand_BN[0][0]          
__________________________________________________________________________________________________
block_2_depthwise (DepthwiseCon (None, None, None, 1 1296        block_2_expand_relu[0][0]        
__________________________________________________________________________________________________
block_2_depthwise_BN (BatchNorm (None, None, None, 1 576         block_2_depthwise[0][0]          
__________________________________________________________________________________________________
block_2_depthwise_relu (ReLU)   (None, None, None, 1 0           block_2_depthwise_BN[0][0]       
__________________________________________________________________________________________________
block_2_project (Conv2D)        (None, None, None, 2 3456        block_2_depthwise_relu[0][0]     
__________________________________________________________________________________________________
block_2_project_BN (BatchNormal (None, None, None, 2 96          block_2_project[0][0]            
__________________________________________________________________________________________________
block_2_add (Add)               (None, None, None, 2 0           block_1_project_BN[0][0]         
                                                                 block_2_project_BN[0][0]         
__________________________________________________________________________________________________
block_3_expand (Conv2D)         (None, None, None, 1 3456        block_2_add[0][0]                
__________________________________________________________________________________________________
block_3_expand_BN (BatchNormali (None, None, None, 1 576         block_3_expand[0][0]             
__________________________________________________________________________________________________
block_3_expand_relu (ReLU)      (None, None, None, 1 0           block_3_expand_BN[0][0]          
__________________________________________________________________________________________________
block_3_pad (ZeroPadding2D)     (None, None, None, 1 0           block_3_expand_relu[0][0]        
__________________________________________________________________________________________________
block_3_depthwise (DepthwiseCon (None, None, None, 1 1296        block_3_pad[0][0]                
__________________________________________________________________________________________________
block_3_depthwise_BN (BatchNorm (None, None, None, 1 576         block_3_depthwise[0][0]          
__________________________________________________________________________________________________
block_3_depthwise_relu (ReLU)   (None, None, None, 1 0           block_3_depthwise_BN[0][0]       
__________________________________________________________________________________________________
block_3_project (Conv2D)        (None, None, None, 3 4608        block_3_depthwise_relu[0][0]     
__________________________________________________________________________________________________
block_3_project_BN (BatchNormal (None, None, None, 3 128         block_3_project[0][0]            
__________________________________________________________________________________________________
block_4_expand (Conv2D)         (None, None, None, 1 6144        block_3_project_BN[0][0]         
__________________________________________________________________________________________________
block_4_expand_BN (BatchNormali (None, None, None, 1 768         block_4_expand[0][0]             
__________________________________________________________________________________________________
block_4_expand_relu (ReLU)      (None, None, None, 1 0           block_4_expand_BN[0][0]          
__________________________________________________________________________________________________
block_4_depthwise (DepthwiseCon (None, None, None, 1 1728        block_4_expand_relu[0][0]        
__________________________________________________________________________________________________
block_4_depthwise_BN (BatchNorm (None, None, None, 1 768         block_4_depthwise[0][0]          
__________________________________________________________________________________________________
block_4_depthwise_relu (ReLU)   (None, None, None, 1 0           block_4_depthwise_BN[0][0]       
__________________________________________________________________________________________________
block_4_project (Conv2D)        (None, None, None, 3 6144        block_4_depthwise_relu[0][0]     
__________________________________________________________________________________________________
block_4_project_BN (BatchNormal (None, None, None, 3 128         block_4_project[0][0]            
__________________________________________________________________________________________________
block_4_add (Add)               (None, None, None, 3 0           block_3_project_BN[0][0]         
                                                                 block_4_project_BN[0][0]         
__________________________________________________________________________________________________
block_5_expand (Conv2D)         (None, None, None, 1 6144        block_4_add[0][0]                
__________________________________________________________________________________________________
block_5_expand_BN (BatchNormali (None, None, None, 1 768         block_5_expand[0][0]             
__________________________________________________________________________________________________
block_5_expand_relu (ReLU)      (None, None, None, 1 0           block_5_expand_BN[0][0]          
__________________________________________________________________________________________________
block_5_depthwise (DepthwiseCon (None, None, None, 1 1728        block_5_expand_relu[0][0]        
__________________________________________________________________________________________________
block_5_depthwise_BN (BatchNorm (None, None, None, 1 768         block_5_depthwise[0][0]          
__________________________________________________________________________________________________
block_5_depthwise_relu (ReLU)   (None, None, None, 1 0           block_5_depthwise_BN[0][0]       
__________________________________________________________________________________________________
block_5_project (Conv2D)        (None, None, None, 3 6144        block_5_depthwise_relu[0][0]     
__________________________________________________________________________________________________
block_5_project_BN (BatchNormal (None, None, None, 3 128         block_5_project[0][0]            
__________________________________________________________________________________________________
block_5_add (Add)               (None, None, None, 3 0           block_4_add[0][0]                
                                                                 block_5_project_BN[0][0]         
__________________________________________________________________________________________________
block_6_expand (Conv2D)         (None, None, None, 1 6144        block_5_add[0][0]                
__________________________________________________________________________________________________
block_6_expand_BN (BatchNormali (None, None, None, 1 768         block_6_expand[0][0]             
__________________________________________________________________________________________________
block_6_expand_relu (ReLU)      (None, None, None, 1 0           block_6_expand_BN[0][0]          
__________________________________________________________________________________________________
block_6_pad (ZeroPadding2D)     (None, None, None, 1 0           block_6_expand_relu[0][0]        
__________________________________________________________________________________________________
block_6_depthwise (DepthwiseCon (None, None, None, 1 1728        block_6_pad[0][0]                
__________________________________________________________________________________________________
block_6_depthwise_BN (BatchNorm (None, None, None, 1 768         block_6_depthwise[0][0]          
__________________________________________________________________________________________________
block_6_depthwise_relu (ReLU)   (None, None, None, 1 0           block_6_depthwise_BN[0][0]       
__________________________________________________________________________________________________
block_6_project (Conv2D)        (None, None, None, 6 12288       block_6_depthwise_relu[0][0]     
__________________________________________________________________________________________________
block_6_project_BN (BatchNormal (None, None, None, 6 256         block_6_project[0][0]            
__________________________________________________________________________________________________
block_7_expand (Conv2D)         (None, None, None, 3 24576       block_6_project_BN[0][0]         
__________________________________________________________________________________________________
block_7_expand_BN (BatchNormali (None, None, None, 3 1536        block_7_expand[0][0]             
__________________________________________________________________________________________________
block_7_expand_relu (ReLU)      (None, None, None, 3 0           block_7_expand_BN[0][0]          
__________________________________________________________________________________________________
block_7_depthwise (DepthwiseCon (None, None, None, 3 3456        block_7_expand_relu[0][0]        
__________________________________________________________________________________________________
block_7_depthwise_BN (BatchNorm (None, None, None, 3 1536        block_7_depthwise[0][0]          
__________________________________________________________________________________________________
block_7_depthwise_relu (ReLU)   (None, None, None, 3 0           block_7_depthwise_BN[0][0]       
__________________________________________________________________________________________________
block_7_project (Conv2D)        (None, None, None, 6 24576       block_7_depthwise_relu[0][0]     
__________________________________________________________________________________________________
block_7_project_BN (BatchNormal (None, None, None, 6 256         block_7_project[0][0]            
__________________________________________________________________________________________________
block_7_add (Add)               (None, None, None, 6 0           block_6_project_BN[0][0]         
                                                                 block_7_project_BN[0][0]         
__________________________________________________________________________________________________
block_8_expand (Conv2D)         (None, None, None, 3 24576       block_7_add[0][0]                
__________________________________________________________________________________________________
block_8_expand_BN (BatchNormali (None, None, None, 3 1536        block_8_expand[0][0]             
__________________________________________________________________________________________________
block_8_expand_relu (ReLU)      (None, None, None, 3 0           block_8_expand_BN[0][0]          
__________________________________________________________________________________________________
block_8_depthwise (DepthwiseCon (None, None, None, 3 3456        block_8_expand_relu[0][0]        
__________________________________________________________________________________________________
block_8_depthwise_BN (BatchNorm (None, None, None, 3 1536        block_8_depthwise[0][0]          
__________________________________________________________________________________________________
block_8_depthwise_relu (ReLU)   (None, None, None, 3 0           block_8_depthwise_BN[0][0]       
__________________________________________________________________________________________________
block_8_project (Conv2D)        (None, None, None, 6 24576       block_8_depthwise_relu[0][0]     
__________________________________________________________________________________________________
block_8_project_BN (BatchNormal (None, None, None, 6 256         block_8_project[0][0]            
__________________________________________________________________________________________________
block_8_add (Add)               (None, None, None, 6 0           block_7_add[0][0]                
                                                                 block_8_project_BN[0][0]         
__________________________________________________________________________________________________
block_9_expand (Conv2D)         (None, None, None, 3 24576       block_8_add[0][0]                
__________________________________________________________________________________________________
block_9_expand_BN (BatchNormali (None, None, None, 3 1536        block_9_expand[0][0]             
__________________________________________________________________________________________________
block_9_expand_relu (ReLU)      (None, None, None, 3 0           block_9_expand_BN[0][0]          
__________________________________________________________________________________________________
block_9_depthwise (DepthwiseCon (None, None, None, 3 3456        block_9_expand_relu[0][0]        
__________________________________________________________________________________________________
block_9_depthwise_BN (BatchNorm (None, None, None, 3 1536        block_9_depthwise[0][0]          
__________________________________________________________________________________________________
block_9_depthwise_relu (ReLU)   (None, None, None, 3 0           block_9_depthwise_BN[0][0]       
__________________________________________________________________________________________________
block_9_project (Conv2D)        (None, None, None, 6 24576       block_9_depthwise_relu[0][0]     
__________________________________________________________________________________________________
block_9_project_BN (BatchNormal (None, None, None, 6 256         block_9_project[0][0]            
__________________________________________________________________________________________________
block_9_add (Add)               (None, None, None, 6 0           block_8_add[0][0]                
                                                                 block_9_project_BN[0][0]         
__________________________________________________________________________________________________
block_10_expand (Conv2D)        (None, None, None, 3 24576       block_9_add[0][0]                
__________________________________________________________________________________________________
block_10_expand_BN (BatchNormal (None, None, None, 3 1536        block_10_expand[0][0]            
__________________________________________________________________________________________________
block_10_expand_relu (ReLU)     (None, None, None, 3 0           block_10_expand_BN[0][0]         
__________________________________________________________________________________________________
block_10_depthwise (DepthwiseCo (None, None, None, 3 3456        block_10_expand_relu[0][0]       
__________________________________________________________________________________________________
block_10_depthwise_BN (BatchNor (None, None, None, 3 1536        block_10_depthwise[0][0]         
__________________________________________________________________________________________________
block_10_depthwise_relu (ReLU)  (None, None, None, 3 0           block_10_depthwise_BN[0][0]      
__________________________________________________________________________________________________
block_10_project (Conv2D)       (None, None, None, 9 36864       block_10_depthwise_relu[0][0]    
__________________________________________________________________________________________________
block_10_project_BN (BatchNorma (None, None, None, 9 384         block_10_project[0][0]           
__________________________________________________________________________________________________
block_11_expand (Conv2D)        (None, None, None, 5 55296       block_10_project_BN[0][0]        
__________________________________________________________________________________________________
block_11_expand_BN (BatchNormal (None, None, None, 5 2304        block_11_expand[0][0]            
__________________________________________________________________________________________________
block_11_expand_relu (ReLU)     (None, None, None, 5 0           block_11_expand_BN[0][0]         
__________________________________________________________________________________________________
block_11_depthwise (DepthwiseCo (None, None, None, 5 5184        block_11_expand_relu[0][0]       
__________________________________________________________________________________________________
block_11_depthwise_BN (BatchNor (None, None, None, 5 2304        block_11_depthwise[0][0]         
__________________________________________________________________________________________________
block_11_depthwise_relu (ReLU)  (None, None, None, 5 0           block_11_depthwise_BN[0][0]      
__________________________________________________________________________________________________
block_11_project (Conv2D)       (None, None, None, 9 55296       block_11_depthwise_relu[0][0]    
__________________________________________________________________________________________________
block_11_project_BN (BatchNorma (None, None, None, 9 384         block_11_project[0][0]           
__________________________________________________________________________________________________
block_11_add (Add)              (None, None, None, 9 0           block_10_project_BN[0][0]        
                                                                 block_11_project_BN[0][0]        
__________________________________________________________________________________________________
block_12_expand (Conv2D)        (None, None, None, 5 55296       block_11_add[0][0]               
__________________________________________________________________________________________________
block_12_expand_BN (BatchNormal (None, None, None, 5 2304        block_12_expand[0][0]            
__________________________________________________________________________________________________
block_12_expand_relu (ReLU)     (None, None, None, 5 0           block_12_expand_BN[0][0]         
__________________________________________________________________________________________________
block_12_depthwise (DepthwiseCo (None, None, None, 5 5184        block_12_expand_relu[0][0]       
__________________________________________________________________________________________________
block_12_depthwise_BN (BatchNor (None, None, None, 5 2304        block_12_depthwise[0][0]         
__________________________________________________________________________________________________
block_12_depthwise_relu (ReLU)  (None, None, None, 5 0           block_12_depthwise_BN[0][0]      
__________________________________________________________________________________________________
block_12_project (Conv2D)       (None, None, None, 9 55296       block_12_depthwise_relu[0][0]    
__________________________________________________________________________________________________
block_12_project_BN (BatchNorma (None, None, None, 9 384         block_12_project[0][0]           
__________________________________________________________________________________________________
block_12_add (Add)              (None, None, None, 9 0           block_11_add[0][0]               
                                                                 block_12_project_BN[0][0]        
__________________________________________________________________________________________________
block_13_expand (Conv2D)        (None, None, None, 5 55296       block_12_add[0][0]               
__________________________________________________________________________________________________
block_13_expand_BN (BatchNormal (None, None, None, 5 2304        block_13_expand[0][0]            
__________________________________________________________________________________________________
block_13_expand_relu (ReLU)     (None, None, None, 5 0           block_13_expand_BN[0][0]         
__________________________________________________________________________________________________
block_13_pad (ZeroPadding2D)    (None, None, None, 5 0           block_13_expand_relu[0][0]       
__________________________________________________________________________________________________
block_13_depthwise (DepthwiseCo (None, None, None, 5 5184        block_13_pad[0][0]               
__________________________________________________________________________________________________
block_13_depthwise_BN (BatchNor (None, None, None, 5 2304        block_13_depthwise[0][0]         
__________________________________________________________________________________________________
block_13_depthwise_relu (ReLU)  (None, None, None, 5 0           block_13_depthwise_BN[0][0]      
__________________________________________________________________________________________________
block_13_project (Conv2D)       (None, None, None, 1 92160       block_13_depthwise_relu[0][0]    
__________________________________________________________________________________________________
block_13_project_BN (BatchNorma (None, None, None, 1 640         block_13_project[0][0]           
__________________________________________________________________________________________________
block_14_expand (Conv2D)        (None, None, None, 9 153600      block_13_project_BN[0][0]        
__________________________________________________________________________________________________
block_14_expand_BN (BatchNormal (None, None, None, 9 3840        block_14_expand[0][0]            
__________________________________________________________________________________________________
block_14_expand_relu (ReLU)     (None, None, None, 9 0           block_14_expand_BN[0][0]         
__________________________________________________________________________________________________
block_14_depthwise (DepthwiseCo (None, None, None, 9 8640        block_14_expand_relu[0][0]       
__________________________________________________________________________________________________
block_14_depthwise_BN (BatchNor (None, None, None, 9 3840        block_14_depthwise[0][0]         
__________________________________________________________________________________________________
block_14_depthwise_relu (ReLU)  (None, None, None, 9 0           block_14_depthwise_BN[0][0]      
__________________________________________________________________________________________________
block_14_project (Conv2D)       (None, None, None, 1 153600      block_14_depthwise_relu[0][0]    
__________________________________________________________________________________________________
block_14_project_BN (BatchNorma (None, None, None, 1 640         block_14_project[0][0]           
__________________________________________________________________________________________________
block_14_add (Add)              (None, None, None, 1 0           block_13_project_BN[0][0]        
                                                                 block_14_project_BN[0][0]        
__________________________________________________________________________________________________
block_15_expand (Conv2D)        (None, None, None, 9 153600      block_14_add[0][0]               
__________________________________________________________________________________________________
block_15_expand_BN (BatchNormal (None, None, None, 9 3840        block_15_expand[0][0]            
__________________________________________________________________________________________________
block_15_expand_relu (ReLU)     (None, None, None, 9 0           block_15_expand_BN[0][0]         
__________________________________________________________________________________________________
block_15_depthwise (DepthwiseCo (None, None, None, 9 8640        block_15_expand_relu[0][0]       
__________________________________________________________________________________________________
block_15_depthwise_BN (BatchNor (None, None, None, 9 3840        block_15_depthwise[0][0]         
__________________________________________________________________________________________________
block_15_depthwise_relu (ReLU)  (None, None, None, 9 0           block_15_depthwise_BN[0][0]      
__________________________________________________________________________________________________
block_15_project (Conv2D)       (None, None, None, 1 153600      block_15_depthwise_relu[0][0]    
__________________________________________________________________________________________________
block_15_project_BN (BatchNorma (None, None, None, 1 640         block_15_project[0][0]           
__________________________________________________________________________________________________
block_15_add (Add)              (None, None, None, 1 0           block_14_add[0][0]               
                                                                 block_15_project_BN[0][0]        
__________________________________________________________________________________________________
block_16_expand (Conv2D)        (None, None, None, 9 153600      block_15_add[0][0]               
__________________________________________________________________________________________________
block_16_expand_BN (BatchNormal (None, None, None, 9 3840        block_16_expand[0][0]            
__________________________________________________________________________________________________
block_16_expand_relu (ReLU)     (None, None, None, 9 0           block_16_expand_BN[0][0]         
__________________________________________________________________________________________________
block_16_depthwise (DepthwiseCo (None, None, None, 9 8640        block_16_expand_relu[0][0]       
__________________________________________________________________________________________________
block_16_depthwise_BN (BatchNor (None, None, None, 9 3840        block_16_depthwise[0][0]         
__________________________________________________________________________________________________
block_16_depthwise_relu (ReLU)  (None, None, None, 9 0           block_16_depthwise_BN[0][0]      
__________________________________________________________________________________________________
block_16_project (Conv2D)       (None, None, None, 3 307200      block_16_depthwise_relu[0][0]    
__________________________________________________________________________________________________
block_16_project_BN (BatchNorma (None, None, None, 3 1280        block_16_project[0][0]           
__________________________________________________________________________________________________
Conv_1 (Conv2D)                 (None, None, None, 1 409600      block_16_project_BN[0][0]        
__________________________________________________________________________________________________
Conv_1_bn (BatchNormalization)  (None, None, None, 1 5120        Conv_1[0][0]                     
__________________________________________________________________________________________________
out_relu (ReLU)                 (None, None, None, 1 0           Conv_1_bn[0][0]                  
==================================================================================================
Total params: 2,257,984
Trainable params: 2,223,872
Non-trainable params: 34,112
__________________________________________________________________________________________________

im = tf.keras.preprocessing.image.load_img('ade.jpg')

np.array(im).shape
# (960, 1280, 3)

plt.imshow(im)

# base_model은 input shape이 224,224,3을 받도록 만들어져 있지만 
# 크기를 고정하지 않았기 때문에 input data로 들어갈 수 있다
base_model(np.array(im)[np.newaxis])

<tf.Tensor: shape=(1, 30, 40, 1280), dtype=float32, numpy=
array([[[[0.        , 0.        , 0.        , ..., 0.        ,
          0.        , 0.        ],
         [0.        , 0.        , 0.        , ..., 0.        ,
          0.        , 0.        ],
         [0.        , 0.        , 0.        , ..., 0.        ,
          0.        , 0.        ],
         ...,
         [0.        , 0.        , 0.        , ..., 0.        ,
          0.        , 0.        ],
         [0.        , 0.        , 0.        , ..., 0.        ,
          0.        , 0.        ],
         [0.        , 0.        , 0.        , ..., 0.        ,
          0.        , 0.        ]],

        [[0.        , 0.        , 0.        , ..., 0.        ,
          0.        , 0.02680304],
         [0.        , 0.        , 0.        , ..., 0.        ,
          2.0374901 , 1.3772459 ],
         [0.        , 0.        , 0.        , ..., 0.        ,
          1.1740499 , 0.43281242],
         ...,
         [0.        , 0.        , 0.        , ..., 0.        ,
          0.        , 0.        ],
         [0.        , 0.        , 0.        , ..., 0.        ,
          0.        , 0.        ],
         [0.        , 0.        , 0.        , ..., 0.        ,
          0.        , 0.        ]],

        [[0.        , 0.7047172 , 0.        , ..., 0.        ,
          1.3220952 , 2.8031015 ],
         [0.        , 0.1590641 , 0.        , ..., 0.        ,
          2.0702174 , 3.4394553 ],
         [0.        , 0.        , 0.        , ..., 0.        ,
          1.5714682 , 2.23192   ],
         ...,
         [0.        , 0.        , 0.        , ..., 0.        ,
          0.        , 0.        ],
         [0.        , 0.        , 0.        , ..., 0.        ,
          0.        , 0.3797317 ],
         [0.        , 0.        , 0.        , ..., 0.        ,
          0.        , 0.        ]],

        ...,

        [[0.        , 0.        , 0.        , ..., 0.        ,
          0.        , 0.        ],
         [0.        , 0.        , 0.        , ..., 0.        ,
          0.        , 0.        ],
         [0.        , 0.        , 0.        , ..., 0.        ,
          0.        , 0.        ],
         ...,
         [0.        , 0.        , 0.        , ..., 0.        ,
          0.        , 0.        ],
         [0.        , 0.        , 0.        , ..., 0.        ,
          0.        , 0.6268955 ],
         [0.        , 0.        , 0.        , ..., 0.        ,
          0.        , 0.09846598]],

        [[0.        , 0.        , 0.        , ..., 0.        ,
          0.        , 0.        ],
         [0.70790523, 0.28814578, 0.        , ..., 0.        ,
          0.        , 0.        ],
         [0.        , 0.20785622, 0.        , ..., 0.        ,
          0.        , 0.        ],
         ...,
         [0.        , 0.        , 0.        , ..., 0.        ,
          0.        , 0.        ],
         [0.        , 0.        , 0.        , ..., 0.        ,
          0.        , 0.        ],
         [0.        , 0.        , 0.        , ..., 0.        ,
          0.        , 0.        ]],

        [[0.        , 0.        , 0.        , ..., 0.        ,
          0.        , 0.        ],
         [0.        , 0.        , 0.        , ..., 0.        ,
          0.        , 0.        ],
         [0.        , 0.        , 0.        , ..., 0.        ,
          0.        , 0.        ],
         ...,
         [0.        , 0.        , 0.        , ..., 0.        ,
          0.        , 0.        ],
         [0.        , 0.        , 0.        , ..., 0.        ,
          0.        , 0.        ],
         [0.        , 0.        , 0.        , ..., 0.        ,
          0.        , 0.        ]]]], dtype=float32)>

# input_shape 고정 / mobilenet(pre-trained model)은 데이터가 적을 때 그나마 괜찮은 성능을 보여준다
base_model = tf.keras.applications.MobileNetV2(input_shape=(128,128,3), include_top=False)

base_model(np.array(im)[np.newaxis]) # input_shape크기를 고정했기 때문에 입력값으로 들어가지 않는다

layer_names = [
    'block_1_expand_relu', # 64x64
    'block_3_expand_relu', # 32x32
    'block_6_expand_relu', # 16x16
    'block_13_expand_relu', # 8x8
    'block_16_project', # 4x4
]
base_model_outputs = [base_model.get_layer(name).output for name in layer_names]

base_model_outputs

[<KerasTensor: shape=(None, 64, 64, 96) dtype=float32 (created by layer 'block_1_expand_relu')>,
 <KerasTensor: shape=(None, 32, 32, 144) dtype=float32 (created by layer 'block_3_expand_relu')>,
 <KerasTensor: shape=(None, 16, 16, 192) dtype=float32 (created by layer 'block_6_expand_relu')>,
 <KerasTensor: shape=(None, 8, 8, 576) dtype=float32 (created by layer 'block_13_expand_relu')>,
 <KerasTensor: shape=(None, 4, 4, 320) dtype=float32 (created by layer 'block_16_project')>]

down_stack = tf.keras.Model(inputs=base_model.input, outputs=base_model_outputs)
down_stack.trainable = False

Concatenate vs Add

- Concatenate는 구조를 더하는 것 (구조를 더하면 정보가 그대로 유지된다)
- Add는 값을 더하는 것

GAN(Generative Adversarial Network)

GAN 구조

Generator(생성자)와 Discriminator (판별자) 두 개의 모델이 동시에 적대적인 과정으로 학습한다
생성자 G는 실제 데이터 분포를 학습하고, 판별자 D는 원래의 데이터인지 생성자로부터 생성이 된 것인지 구분한다
생성자 G의 학습 과정은 이미지를 잘 생성해서 속일 확률을 높이고 판별자 D가 제대로 구분하는 확률을 높이는 방향으로 학습을 진행한다

GAN은 진짜 같은 가짜 데이터를 generator가 생성하고 진짜 데이터와 비교하여 discriminator가 어느 것이 진짜인지 판별하며 학습해 나가는 모델이다

GAN의 종류

(X_train, y_train), (X_test, y_test) = tf.keras.datasets.mnist.load_data()

X_train = X_train.reshape(-1,28*28) / 255
X_test = X_test.reshape(-1,28*28) / 255

input_ = tf.keras.Input((10,))
x = tf.keras.layers.Dense(128)(input_)
x = tf.keras.layers.LeakyReLU()(x)
x = tf.keras.layers.Dense(256)(x)
x = tf.keras.layers.LeakyReLU()(x)
x = tf.keras.layers.Dense(512)(x)
x = tf.keras.layers.LeakyReLU()(x)
x = tf.keras.layers.Dense(28*28, activation='sigmoid')(x)

generator = tf.keras.models.Model(input_, x)

input_ = tf.keras.Input((784,))
x = tf.keras.layers.Dense(1024)(input_)
x = tf.keras.layers.LeakyReLU()(x)
x = tf.keras.layers.Dense(512)(x)
x = tf.keras.layers.LeakyReLU()(x)
x = tf.keras.layers.Dense(256)(x)
x = tf.keras.layers.LeakyReLU()(x)
x = tf.keras.layers.Dense(1, activation='sigmoid')(x)

discriminator = tf.keras.models.Model(input_, x)
discriminator.compile(loss='binary_crossentropy',optimizer='adam')

discriminator.trainable = False

gan_input = tf.keras.Input((10,))
x = generator(gan_input)
output = discriminator(x)

GAN = tf.keras.models.Model(gan_input, output)
GAN.compile(loss='binary_crossentropy',optimizer='adam')

GAN.summary()

Model: "model_2"
_________________________________________________________________
Layer (type)                 Output Shape              Param #   
=================================================================
input_3 (InputLayer)         [(None, 10)]              0         
_________________________________________________________________
model (Functional)           (None, 784)               568208    
_________________________________________________________________
model_1 (Functional)         (None, 1)                 1460225   
=================================================================
Total params: 2,028,433
Trainable params: 568,208
Non-trainable params: 1,460,225
_________________________________________________________________

def get_batch(data, batch_size=32):
    batches = []
    for i in range(int(data.shape[0]//batch_size)):
        batch = data[i*batch_size: (i+1)*batch_size]
        batches.append(batch)
    return np.asarray(batches)
    

get_batch(X_train, 100).shape
#(600, 100, 784)

d_losses = []
g_losses = []
for i in range(1,11):
    for j in get_batch(X_train):
        input_noise = np.random.uniform(-1,1, size=[32,10])
        fake = generator.predict(input_noise)
        x_dis = np.concatenate([j, fake])
        y_dis = np.zeros(2*32)
        y_dis[:32] = 1
        
        discriminator.trainable = True 
        d_loss = discriminator.train_on_batch(x_dis, y_dis) # fit 
        discriminator.trainable = False 
        
        noise = np.random.uniform(-1,1,size=[32,10])
        y_gan = np.ones(32)
        g_loss = GAN.train_on_batch(noise, y_gan)
        
    d_losses.append(d_loss)
    g_losses.append(g_loss)

fake = generator.predict(noise).reshape(-1, 28,28)

plt.imshow(fake[0]) # 한 번 학습 했을 때

plt.imshow(fake[1]) # 한 번 학습 했을 때

plt.imshow(fake[0]) # 두 번 학습 했을 때

plt.imshow(fake[1]) # 두 번 학습 했을 때

Train_on_batch

Train_on_batch는 고정된 batch size를 신경쓰지 않고 현재 전달받은 입력 데이터를 활용하여 weight를 업데이트한다
GAN에서는 매번 Generator가 새로운 fake image를 만들기 때문에 epoch마다 새로운 데이터를 넘겨줘야 한다
따라서 train_on_batch를 사용하는 것이 좋다
또한 pre-trained 모델을 단일 배치로 학습시켜야 할 때 train_on_batch를 사용하는 것이 좋다

저작자표시 비영리

'Computer_Science > Visual Intelligence' 카테고리의 다른 글

39일차 - 논문 수업 (GAN) (0)	2021.11.16
37일차 - 논문 수업 (Segmentation) (0)	2021.11.16
36일차 - 논문 수업 (Segmentation) (0)	2021.11.16
35일차 - 논문 수업 (Segmentation) (0)	2021.11.01
34일차 - 논문 수업 (VAE) (0)	2021.11.01

37일차 - 논문 수업 (Segmentation)

2021. 11. 16. 18:56

728x90

Segmentation

이미지에서 점 단위로 예측하는 기법

확인해야 할 포인트
1. 어떻게 한점 한점을 예측할지
2. 원본의 크기와 같아야 한다
- 입력된 이미지를 넣었을 때 원본과 같은 크기로 나와야 한다

종류

1. Semantic segmentation
- 이미지 내에 있는 물체들을 의미 있는 단위로 분할한다
- 이미지의 각 픽셀이 어느 클래스에 속하는지 예측한다
- 동일한 객체들끼리는 한 번에 Masking을 수행한다
- 대표적으로 FCN이 있다
2. Instance segmentation
- 동일한 객체들이여도 개별 Object로 간주하고 Masking을 수행한다
- 대표적으로 Mask R-CNN이 있다

FCN(Fully Convolutional Networks)

AlexNet의 구조에서 Dense layer 부분을 Convolution layer로 변형한 모델 (convolutionalization)

Semantic segmentation을 하기 위해서는 AlexNet 구조에서 FC layer를 변형 해야 하는 이유가 있다

1. FC layer를 사용하면 이미지의 위치 정보가 사라진다
2. FC layer를 사용하면 입력 이미지의 크기가 고정된다

따라서 semantic segmenation을 하기 위해서 마지막 FC layer를 convolution layer로
변형 함으로써 위치 정보를 보존하고 이미지 크기로 부터 유연할 수 있다

Skip combining

이전의 정보를 더함으로써 복원할때 활용한다
이전의 layer는 현재의 layer가 가지는 정보보다 디테일한 정보를 가지고 있기 때문에 복원하는데 도움이 되기 때문이다

U-net

Instance segmentation 방법

U-net 구조

U-Net은 이미지의 전반적인 컨텍스트 정보를 얻기 위한 네트워크와 정확한 지역화(Localization)를 위한 네트워크가 대칭 형태로 구성되어 있다

Contracting path의 최종 feature map으로 부터 높은 해상도의 결과를 얻기 위해 upsampling을 진행한다
coarse map -> dense prediction

Overlap-Tile Input

이미지를 타일로 나누어서 입력으로 사용한다
파란 영역의 이미지를 입력하면 노란 영역의 segmentation 결과를 얻는다

다음 tile에 대한 segmentation을 얻기 위해서는 이전 입력의 일부분이 포함되어 있어야 한다

이미지의 경계 부분 픽셀에 대한 segmentation을 위해 0이나 임의의 패딩값을 사용하는 대신
이미지 경계 부분의 미러링을 이용한 Extrapolation 기법을 사용한다

U-net 구현하기

import tensorflow as tf 
import tensorflow_datasets as tfds
import matplotlib.pyplot as plt

dataset, info = tfds.load('oxford_iiit_pet:3.*.*', with_info=True) # 37마리 애완동물 6000장 이미지 

dataset['train'] # cpu, gpu 둘다 활용하여 효율적 연산 가능한 데이터 구조 
# <PrefetchDataset shapes: {file_name: (), image: (None, None, 3), label: (), segmentation_mask: (None, None, 1), species: ()}, types: {file_name: tf.string, image: tf.uint8, label: tf.int64, segmentation_mask: tf.uint8, species: tf.int64}>

dataset['test']

# <PrefetchDataset shapes: {file_name: (), image: (None, None, 3), label: (), segmentation_mask: (None, None, 1), species: ()}, types: {file_name: tf.string, image: tf.uint8, label: tf.int64, segmentation_mask: tf.uint8, species: tf.int64}>

for i in dataset['train'].take(1):
    image = i['image']
    mask = i['segmentation_mask']

fig, ax = plt.subplots(1,2)
ax[0].imshow(image)
ax[1].imshow(mask, cmap='gray')

backbone = tf.keras.applications.MobileNetV2(input_shape=(128,128,3), include_top=False)

backbone.summary()
Model: "mobilenetv2_1.00_128"
__________________________________________________________________________________________________
Layer (type)                    Output Shape         Param #     Connected to                     
==================================================================================================
input_1 (InputLayer)            [(None, 128, 128, 3) 0                                            
__________________________________________________________________________________________________
Conv1 (Conv2D)                  (None, 64, 64, 32)   864         input_1[0][0]                    
__________________________________________________________________________________________________
bn_Conv1 (BatchNormalization)   (None, 64, 64, 32)   128         Conv1[0][0]                      
__________________________________________________________________________________________________
Conv1_relu (ReLU)               (None, 64, 64, 32)   0           bn_Conv1[0][0]                   
__________________________________________________________________________________________________
expanded_conv_depthwise (Depthw (None, 64, 64, 32)   288         Conv1_relu[0][0]                 
__________________________________________________________________________________________________
expanded_conv_depthwise_BN (Bat (None, 64, 64, 32)   128         expanded_conv_depthwise[0][0]    
__________________________________________________________________________________________________
expanded_conv_depthwise_relu (R (None, 64, 64, 32)   0           expanded_conv_depthwise_BN[0][0] 
__________________________________________________________________________________________________
expanded_conv_project (Conv2D)  (None, 64, 64, 16)   512         expanded_conv_depthwise_relu[0][0
__________________________________________________________________________________________________
expanded_conv_project_BN (Batch (None, 64, 64, 16)   64          expanded_conv_project[0][0]      
__________________________________________________________________________________________________
block_1_expand (Conv2D)         (None, 64, 64, 96)   1536        expanded_conv_project_BN[0][0]   
__________________________________________________________________________________________________
block_1_expand_BN (BatchNormali (None, 64, 64, 96)   384         block_1_expand[0][0]             
__________________________________________________________________________________________________
block_1_expand_relu (ReLU)      (None, 64, 64, 96)   0           block_1_expand_BN[0][0]          
__________________________________________________________________________________________________
block_1_pad (ZeroPadding2D)     (None, 65, 65, 96)   0           block_1_expand_relu[0][0]        
__________________________________________________________________________________________________
block_1_depthwise (DepthwiseCon (None, 32, 32, 96)   864         block_1_pad[0][0]                
__________________________________________________________________________________________________
block_1_depthwise_BN (BatchNorm (None, 32, 32, 96)   384         block_1_depthwise[0][0]          
__________________________________________________________________________________________________
block_1_depthwise_relu (ReLU)   (None, 32, 32, 96)   0           block_1_depthwise_BN[0][0]       
__________________________________________________________________________________________________
block_1_project (Conv2D)        (None, 32, 32, 24)   2304        block_1_depthwise_relu[0][0]     
__________________________________________________________________________________________________
block_1_project_BN (BatchNormal (None, 32, 32, 24)   96          block_1_project[0][0]            
__________________________________________________________________________________________________
block_2_expand (Conv2D)         (None, 32, 32, 144)  3456        block_1_project_BN[0][0]         
__________________________________________________________________________________________________
block_2_expand_BN (BatchNormali (None, 32, 32, 144)  576         block_2_expand[0][0]             
__________________________________________________________________________________________________
block_2_expand_relu (ReLU)      (None, 32, 32, 144)  0           block_2_expand_BN[0][0]          
__________________________________________________________________________________________________
block_2_depthwise (DepthwiseCon (None, 32, 32, 144)  1296        block_2_expand_relu[0][0]        
__________________________________________________________________________________________________
block_2_depthwise_BN (BatchNorm (None, 32, 32, 144)  576         block_2_depthwise[0][0]          
__________________________________________________________________________________________________
block_2_depthwise_relu (ReLU)   (None, 32, 32, 144)  0           block_2_depthwise_BN[0][0]       
__________________________________________________________________________________________________
block_2_project (Conv2D)        (None, 32, 32, 24)   3456        block_2_depthwise_relu[0][0]     
__________________________________________________________________________________________________
block_2_project_BN (BatchNormal (None, 32, 32, 24)   96          block_2_project[0][0]            
__________________________________________________________________________________________________
block_2_add (Add)               (None, 32, 32, 24)   0           block_1_project_BN[0][0]         
                                                                 block_2_project_BN[0][0]         
__________________________________________________________________________________________________
block_3_expand (Conv2D)         (None, 32, 32, 144)  3456        block_2_add[0][0]                
__________________________________________________________________________________________________
block_3_expand_BN (BatchNormali (None, 32, 32, 144)  576         block_3_expand[0][0]             
__________________________________________________________________________________________________
block_3_expand_relu (ReLU)      (None, 32, 32, 144)  0           block_3_expand_BN[0][0]          
__________________________________________________________________________________________________
block_3_pad (ZeroPadding2D)     (None, 33, 33, 144)  0           block_3_expand_relu[0][0]        
__________________________________________________________________________________________________
block_3_depthwise (DepthwiseCon (None, 16, 16, 144)  1296        block_3_pad[0][0]                
__________________________________________________________________________________________________
block_3_depthwise_BN (BatchNorm (None, 16, 16, 144)  576         block_3_depthwise[0][0]          
__________________________________________________________________________________________________
block_3_depthwise_relu (ReLU)   (None, 16, 16, 144)  0           block_3_depthwise_BN[0][0]       
__________________________________________________________________________________________________
block_3_project (Conv2D)        (None, 16, 16, 32)   4608        block_3_depthwise_relu[0][0]     
__________________________________________________________________________________________________
block_3_project_BN (BatchNormal (None, 16, 16, 32)   128         block_3_project[0][0]            
__________________________________________________________________________________________________
block_4_expand (Conv2D)         (None, 16, 16, 192)  6144        block_3_project_BN[0][0]         
__________________________________________________________________________________________________
block_4_expand_BN (BatchNormali (None, 16, 16, 192)  768         block_4_expand[0][0]             
__________________________________________________________________________________________________
block_4_expand_relu (ReLU)      (None, 16, 16, 192)  0           block_4_expand_BN[0][0]          
__________________________________________________________________________________________________
block_4_depthwise (DepthwiseCon (None, 16, 16, 192)  1728        block_4_expand_relu[0][0]        
__________________________________________________________________________________________________
block_4_depthwise_BN (BatchNorm (None, 16, 16, 192)  768         block_4_depthwise[0][0]          
__________________________________________________________________________________________________
block_4_depthwise_relu (ReLU)   (None, 16, 16, 192)  0           block_4_depthwise_BN[0][0]       
__________________________________________________________________________________________________
block_4_project (Conv2D)        (None, 16, 16, 32)   6144        block_4_depthwise_relu[0][0]     
__________________________________________________________________________________________________
block_4_project_BN (BatchNormal (None, 16, 16, 32)   128         block_4_project[0][0]            
__________________________________________________________________________________________________
block_4_add (Add)               (None, 16, 16, 32)   0           block_3_project_BN[0][0]         
                                                                 block_4_project_BN[0][0]         
__________________________________________________________________________________________________
block_5_expand (Conv2D)         (None, 16, 16, 192)  6144        block_4_add[0][0]                
__________________________________________________________________________________________________
block_5_expand_BN (BatchNormali (None, 16, 16, 192)  768         block_5_expand[0][0]             
__________________________________________________________________________________________________
block_5_expand_relu (ReLU)      (None, 16, 16, 192)  0           block_5_expand_BN[0][0]          
__________________________________________________________________________________________________
block_5_depthwise (DepthwiseCon (None, 16, 16, 192)  1728        block_5_expand_relu[0][0]        
__________________________________________________________________________________________________
block_5_depthwise_BN (BatchNorm (None, 16, 16, 192)  768         block_5_depthwise[0][0]          
__________________________________________________________________________________________________
block_5_depthwise_relu (ReLU)   (None, 16, 16, 192)  0           block_5_depthwise_BN[0][0]       
__________________________________________________________________________________________________
block_5_project (Conv2D)        (None, 16, 16, 32)   6144        block_5_depthwise_relu[0][0]     
__________________________________________________________________________________________________
block_5_project_BN (BatchNormal (None, 16, 16, 32)   128         block_5_project[0][0]            
__________________________________________________________________________________________________
block_5_add (Add)               (None, 16, 16, 32)   0           block_4_add[0][0]                
                                                                 block_5_project_BN[0][0]         
__________________________________________________________________________________________________
block_6_expand (Conv2D)         (None, 16, 16, 192)  6144        block_5_add[0][0]                
__________________________________________________________________________________________________
block_6_expand_BN (BatchNormali (None, 16, 16, 192)  768         block_6_expand[0][0]             
__________________________________________________________________________________________________
block_6_expand_relu (ReLU)      (None, 16, 16, 192)  0           block_6_expand_BN[0][0]          
__________________________________________________________________________________________________
block_6_pad (ZeroPadding2D)     (None, 17, 17, 192)  0           block_6_expand_relu[0][0]        
__________________________________________________________________________________________________
block_6_depthwise (DepthwiseCon (None, 8, 8, 192)    1728        block_6_pad[0][0]                
__________________________________________________________________________________________________
block_6_depthwise_BN (BatchNorm (None, 8, 8, 192)    768         block_6_depthwise[0][0]          
__________________________________________________________________________________________________
block_6_depthwise_relu (ReLU)   (None, 8, 8, 192)    0           block_6_depthwise_BN[0][0]       
__________________________________________________________________________________________________
block_6_project (Conv2D)        (None, 8, 8, 64)     12288       block_6_depthwise_relu[0][0]     
__________________________________________________________________________________________________
block_6_project_BN (BatchNormal (None, 8, 8, 64)     256         block_6_project[0][0]            
__________________________________________________________________________________________________
block_7_expand (Conv2D)         (None, 8, 8, 384)    24576       block_6_project_BN[0][0]         
__________________________________________________________________________________________________
block_7_expand_BN (BatchNormali (None, 8, 8, 384)    1536        block_7_expand[0][0]             
__________________________________________________________________________________________________
block_7_expand_relu (ReLU)      (None, 8, 8, 384)    0           block_7_expand_BN[0][0]          
__________________________________________________________________________________________________
block_7_depthwise (DepthwiseCon (None, 8, 8, 384)    3456        block_7_expand_relu[0][0]        
__________________________________________________________________________________________________
block_7_depthwise_BN (BatchNorm (None, 8, 8, 384)    1536        block_7_depthwise[0][0]          
__________________________________________________________________________________________________
block_7_depthwise_relu (ReLU)   (None, 8, 8, 384)    0           block_7_depthwise_BN[0][0]       
__________________________________________________________________________________________________
block_7_project (Conv2D)        (None, 8, 8, 64)     24576       block_7_depthwise_relu[0][0]     
__________________________________________________________________________________________________
block_7_project_BN (BatchNormal (None, 8, 8, 64)     256         block_7_project[0][0]            
__________________________________________________________________________________________________
block_7_add (Add)               (None, 8, 8, 64)     0           block_6_project_BN[0][0]         
                                                                 block_7_project_BN[0][0]         
__________________________________________________________________________________________________
block_8_expand (Conv2D)         (None, 8, 8, 384)    24576       block_7_add[0][0]                
__________________________________________________________________________________________________
block_8_expand_BN (BatchNormali (None, 8, 8, 384)    1536        block_8_expand[0][0]             
__________________________________________________________________________________________________
block_8_expand_relu (ReLU)      (None, 8, 8, 384)    0           block_8_expand_BN[0][0]          
__________________________________________________________________________________________________
block_8_depthwise (DepthwiseCon (None, 8, 8, 384)    3456        block_8_expand_relu[0][0]        
__________________________________________________________________________________________________
block_8_depthwise_BN (BatchNorm (None, 8, 8, 384)    1536        block_8_depthwise[0][0]          
__________________________________________________________________________________________________
block_8_depthwise_relu (ReLU)   (None, 8, 8, 384)    0           block_8_depthwise_BN[0][0]       
__________________________________________________________________________________________________
block_8_project (Conv2D)        (None, 8, 8, 64)     24576       block_8_depthwise_relu[0][0]     
__________________________________________________________________________________________________
block_8_project_BN (BatchNormal (None, 8, 8, 64)     256         block_8_project[0][0]            
__________________________________________________________________________________________________
block_8_add (Add)               (None, 8, 8, 64)     0           block_7_add[0][0]                
                                                                 block_8_project_BN[0][0]         
__________________________________________________________________________________________________
block_9_expand (Conv2D)         (None, 8, 8, 384)    24576       block_8_add[0][0]                
__________________________________________________________________________________________________
block_9_expand_BN (BatchNormali (None, 8, 8, 384)    1536        block_9_expand[0][0]             
__________________________________________________________________________________________________
block_9_expand_relu (ReLU)      (None, 8, 8, 384)    0           block_9_expand_BN[0][0]          
__________________________________________________________________________________________________
block_9_depthwise (DepthwiseCon (None, 8, 8, 384)    3456        block_9_expand_relu[0][0]        
__________________________________________________________________________________________________
block_9_depthwise_BN (BatchNorm (None, 8, 8, 384)    1536        block_9_depthwise[0][0]          
__________________________________________________________________________________________________
block_9_depthwise_relu (ReLU)   (None, 8, 8, 384)    0           block_9_depthwise_BN[0][0]       
__________________________________________________________________________________________________
block_9_project (Conv2D)        (None, 8, 8, 64)     24576       block_9_depthwise_relu[0][0]     
__________________________________________________________________________________________________
block_9_project_BN (BatchNormal (None, 8, 8, 64)     256         block_9_project[0][0]            
__________________________________________________________________________________________________
block_9_add (Add)               (None, 8, 8, 64)     0           block_8_add[0][0]                
                                                                 block_9_project_BN[0][0]         
__________________________________________________________________________________________________
block_10_expand (Conv2D)        (None, 8, 8, 384)    24576       block_9_add[0][0]                
__________________________________________________________________________________________________
block_10_expand_BN (BatchNormal (None, 8, 8, 384)    1536        block_10_expand[0][0]            
__________________________________________________________________________________________________
block_10_expand_relu (ReLU)     (None, 8, 8, 384)    0           block_10_expand_BN[0][0]         
__________________________________________________________________________________________________
block_10_depthwise (DepthwiseCo (None, 8, 8, 384)    3456        block_10_expand_relu[0][0]       
__________________________________________________________________________________________________
block_10_depthwise_BN (BatchNor (None, 8, 8, 384)    1536        block_10_depthwise[0][0]         
__________________________________________________________________________________________________
block_10_depthwise_relu (ReLU)  (None, 8, 8, 384)    0           block_10_depthwise_BN[0][0]      
__________________________________________________________________________________________________
block_10_project (Conv2D)       (None, 8, 8, 96)     36864       block_10_depthwise_relu[0][0]    
__________________________________________________________________________________________________
block_10_project_BN (BatchNorma (None, 8, 8, 96)     384         block_10_project[0][0]           
__________________________________________________________________________________________________
block_11_expand (Conv2D)        (None, 8, 8, 576)    55296       block_10_project_BN[0][0]        
__________________________________________________________________________________________________
block_11_expand_BN (BatchNormal (None, 8, 8, 576)    2304        block_11_expand[0][0]            
__________________________________________________________________________________________________
block_11_expand_relu (ReLU)     (None, 8, 8, 576)    0           block_11_expand_BN[0][0]         
__________________________________________________________________________________________________
block_11_depthwise (DepthwiseCo (None, 8, 8, 576)    5184        block_11_expand_relu[0][0]       
__________________________________________________________________________________________________
block_11_depthwise_BN (BatchNor (None, 8, 8, 576)    2304        block_11_depthwise[0][0]         
__________________________________________________________________________________________________
block_11_depthwise_relu (ReLU)  (None, 8, 8, 576)    0           block_11_depthwise_BN[0][0]      
__________________________________________________________________________________________________
block_11_project (Conv2D)       (None, 8, 8, 96)     55296       block_11_depthwise_relu[0][0]    
__________________________________________________________________________________________________
block_11_project_BN (BatchNorma (None, 8, 8, 96)     384         block_11_project[0][0]           
__________________________________________________________________________________________________
block_11_add (Add)              (None, 8, 8, 96)     0           block_10_project_BN[0][0]        
                                                                 block_11_project_BN[0][0]        
__________________________________________________________________________________________________
block_12_expand (Conv2D)        (None, 8, 8, 576)    55296       block_11_add[0][0]               
__________________________________________________________________________________________________
block_12_expand_BN (BatchNormal (None, 8, 8, 576)    2304        block_12_expand[0][0]            
__________________________________________________________________________________________________
block_12_expand_relu (ReLU)     (None, 8, 8, 576)    0           block_12_expand_BN[0][0]         
__________________________________________________________________________________________________
block_12_depthwise (DepthwiseCo (None, 8, 8, 576)    5184        block_12_expand_relu[0][0]       
__________________________________________________________________________________________________
block_12_depthwise_BN (BatchNor (None, 8, 8, 576)    2304        block_12_depthwise[0][0]         
__________________________________________________________________________________________________
block_12_depthwise_relu (ReLU)  (None, 8, 8, 576)    0           block_12_depthwise_BN[0][0]      
__________________________________________________________________________________________________
block_12_project (Conv2D)       (None, 8, 8, 96)     55296       block_12_depthwise_relu[0][0]    
__________________________________________________________________________________________________
block_12_project_BN (BatchNorma (None, 8, 8, 96)     384         block_12_project[0][0]           
__________________________________________________________________________________________________
block_12_add (Add)              (None, 8, 8, 96)     0           block_11_add[0][0]               
                                                                 block_12_project_BN[0][0]        
__________________________________________________________________________________________________
block_13_expand (Conv2D)        (None, 8, 8, 576)    55296       block_12_add[0][0]               
__________________________________________________________________________________________________
block_13_expand_BN (BatchNormal (None, 8, 8, 576)    2304        block_13_expand[0][0]            
__________________________________________________________________________________________________
block_13_expand_relu (ReLU)     (None, 8, 8, 576)    0           block_13_expand_BN[0][0]         
__________________________________________________________________________________________________
block_13_pad (ZeroPadding2D)    (None, 9, 9, 576)    0           block_13_expand_relu[0][0]       
__________________________________________________________________________________________________
block_13_depthwise (DepthwiseCo (None, 4, 4, 576)    5184        block_13_pad[0][0]               
__________________________________________________________________________________________________
block_13_depthwise_BN (BatchNor (None, 4, 4, 576)    2304        block_13_depthwise[0][0]         
__________________________________________________________________________________________________
block_13_depthwise_relu (ReLU)  (None, 4, 4, 576)    0           block_13_depthwise_BN[0][0]      
__________________________________________________________________________________________________
block_13_project (Conv2D)       (None, 4, 4, 160)    92160       block_13_depthwise_relu[0][0]    
__________________________________________________________________________________________________
block_13_project_BN (BatchNorma (None, 4, 4, 160)    640         block_13_project[0][0]           
__________________________________________________________________________________________________
block_14_expand (Conv2D)        (None, 4, 4, 960)    153600      block_13_project_BN[0][0]        
__________________________________________________________________________________________________
block_14_expand_BN (BatchNormal (None, 4, 4, 960)    3840        block_14_expand[0][0]            
__________________________________________________________________________________________________
block_14_expand_relu (ReLU)     (None, 4, 4, 960)    0           block_14_expand_BN[0][0]         
__________________________________________________________________________________________________
block_14_depthwise (DepthwiseCo (None, 4, 4, 960)    8640        block_14_expand_relu[0][0]       
__________________________________________________________________________________________________
block_14_depthwise_BN (BatchNor (None, 4, 4, 960)    3840        block_14_depthwise[0][0]         
__________________________________________________________________________________________________
block_14_depthwise_relu (ReLU)  (None, 4, 4, 960)    0           block_14_depthwise_BN[0][0]      
__________________________________________________________________________________________________
block_14_project (Conv2D)       (None, 4, 4, 160)    153600      block_14_depthwise_relu[0][0]    
__________________________________________________________________________________________________
block_14_project_BN (BatchNorma (None, 4, 4, 160)    640         block_14_project[0][0]           
__________________________________________________________________________________________________
block_14_add (Add)              (None, 4, 4, 160)    0           block_13_project_BN[0][0]        
                                                                 block_14_project_BN[0][0]        
__________________________________________________________________________________________________
block_15_expand (Conv2D)        (None, 4, 4, 960)    153600      block_14_add[0][0]               
__________________________________________________________________________________________________
block_15_expand_BN (BatchNormal (None, 4, 4, 960)    3840        block_15_expand[0][0]            
__________________________________________________________________________________________________
block_15_expand_relu (ReLU)     (None, 4, 4, 960)    0           block_15_expand_BN[0][0]         
__________________________________________________________________________________________________
block_15_depthwise (DepthwiseCo (None, 4, 4, 960)    8640        block_15_expand_relu[0][0]       
__________________________________________________________________________________________________
block_15_depthwise_BN (BatchNor (None, 4, 4, 960)    3840        block_15_depthwise[0][0]         
__________________________________________________________________________________________________
block_15_depthwise_relu (ReLU)  (None, 4, 4, 960)    0           block_15_depthwise_BN[0][0]      
__________________________________________________________________________________________________
block_15_project (Conv2D)       (None, 4, 4, 160)    153600      block_15_depthwise_relu[0][0]    
__________________________________________________________________________________________________
block_15_project_BN (BatchNorma (None, 4, 4, 160)    640         block_15_project[0][0]           
__________________________________________________________________________________________________
block_15_add (Add)              (None, 4, 4, 160)    0           block_14_add[0][0]               
                                                                 block_15_project_BN[0][0]        
__________________________________________________________________________________________________
block_16_expand (Conv2D)        (None, 4, 4, 960)    153600      block_15_add[0][0]               
__________________________________________________________________________________________________
block_16_expand_BN (BatchNormal (None, 4, 4, 960)    3840        block_16_expand[0][0]            
__________________________________________________________________________________________________
block_16_expand_relu (ReLU)     (None, 4, 4, 960)    0           block_16_expand_BN[0][0]         
__________________________________________________________________________________________________
block_16_depthwise (DepthwiseCo (None, 4, 4, 960)    8640        block_16_expand_relu[0][0]       
__________________________________________________________________________________________________
block_16_depthwise_BN (BatchNor (None, 4, 4, 960)    3840        block_16_depthwise[0][0]         
__________________________________________________________________________________________________
block_16_depthwise_relu (ReLU)  (None, 4, 4, 960)    0           block_16_depthwise_BN[0][0]      
__________________________________________________________________________________________________
block_16_project (Conv2D)       (None, 4, 4, 320)    307200      block_16_depthwise_relu[0][0]    
__________________________________________________________________________________________________
block_16_project_BN (BatchNorma (None, 4, 4, 320)    1280        block_16_project[0][0]           
__________________________________________________________________________________________________
Conv_1 (Conv2D)                 (None, 4, 4, 1280)   409600      block_16_project_BN[0][0]        
__________________________________________________________________________________________________
Conv_1_bn (BatchNormalization)  (None, 4, 4, 1280)   5120        Conv_1[0][0]                     
__________________________________________________________________________________________________
out_relu (ReLU)                 (None, 4, 4, 1280)   0           Conv_1_bn[0][0]                  
==================================================================================================
Total params: 2,257,984
Trainable params: 2,223,872
Non-trainable params: 34,112
__________________________________________________________________________________________________

from tensorflow.keras.utils import plot_model
plot_model(backbone)

len(backbone.layers)
# 154

layer_names = [
    'block_1_expand_relu', # 64x64
    'block_3_expand_relu', # 32x32
    'block_6_expand_relu', # 16x16
    'block_13_expand_relu', # 8x8
    'block_16_project', # 4x4
]
base_model_outputs = [base_model.get_layer(name).output for name in layer_names]

저작자표시 비영리

'Computer_Science > Visual Intelligence' 카테고리의 다른 글

39일차 - 논문 수업 (GAN) (0)	2021.11.16
38일차 - Style Transfer (0)	2021.11.16
36일차 - 논문 수업 (Segmentation) (0)	2021.11.16
35일차 - 논문 수업 (Segmentation) (0)	2021.11.01
34일차 - 논문 수업 (VAE) (0)	2021.11.01

36일차 - 논문 수업 (Segmentation)

2021. 11. 16. 18:41

728x90

Variational AutoEncoder

Reparametization trick

확률의 문제를 학습의 문제로 변형하기 위한 방법
미분 가능한 식으로 변형하여 backpropagation이 가능하도록 변형한다

import tensorflow as tf 
from tensorflow.keras import backend as K # backend(tensorflow)에 맞추어 low level 데이터 연산을 해주는 라이브러리 
import inspect

print(inspect.getsource(K.random_normal)) # 정규분포로 부터 무작위 샘플을 추출 하는 함수 


@keras_export('keras.backend.random_normal')
@tf.__internal__.dispatch.add_dispatch_support
@doc_controls.do_not_generate_docs
def random_normal(shape, mean=0.0, stddev=1.0, dtype=None, seed=None):
  """Returns a tensor with normal distribution of values.

  It is an alias to `tf.random.normal`.

  Args:
      shape: A tuple of integers, the shape of tensor to create.
      mean: A float, the mean value of the normal distribution to draw samples.
        Default to 0.0.
      stddev: A float, the standard deviation of the normal distribution
        to draw samples. Default to 1.0.
      dtype: `tf.dtypes.DType`, dtype of returned tensor. Default to use Keras
        backend dtype which is float32.
      seed: Integer, random seed. Will use a random numpy integer when not
        specified.

  Returns:
      A tensor with normal distribution of values.

  Example:

  >>> random_normal_tensor = tf.keras.backend.random_normal(shape=(2,3),
  ... mean=0.0, stddev=1.0)
  >>> random_normal_tensor
  <tf.Tensor: shape=(2, 3), dtype=float32, numpy=...,
  dtype=float32)>
  """
  if dtype is None:
    dtype = floatx()
  if seed is None:
    seed = np.random.randint(10e6)
  return tf.random.normal(
      shape, mean=mean, stddev=stddev, dtype=dtype, seed=seed)

tf.keras.losses.kl_divergence

# <function keras.losses.kl_divergence(y_true, y_pred)>

K.random_normal((2,3))

# <tf.Tensor: shape=(2, 3), dtype=float32, numpy=
# array([[-0.03670657,  0.19654608,  1.8087223 ],
#        [-0.5491878 , -1.7109    ,  0.6098757 ]], dtype=float32)>

epsilon = K.random_normal(shape=(K.shape(z_mean)[0], latent_dim), mean=0, stddev=0.1)

random normal 자체는 미분이 가능하지만 random normal 결과는 미분가능하지 않기 때문에 파라미터화 해야 한다

from sklearn.linear_model import LinearRegression, Ridge, Lasso, ElasticNet

Regularizer term

Regularizer term은 λ또는 Regularization Parameter라고도 부른다
Regularizer term은 급격한 변화를 방지하는 역할을 한다. 결국 일반화 된 모델이 만들어지도록 유도한다

Lasso
- L1-norm 패널티를 가진 선형 회귀 방법, overfitting을 방지하는 방법이다
Ridge
- L2-norm 패널티를 가진 선형 회귀 방법, overfitting을 방지하는 방법이다
ElasticNet L1, L2
- L1-norm, L2-norm 패널티를 동시에 가진 선형 회귀 방법, overfitting을 방지하는 방법이다

Lasso

MSE를 최소로 만드는 w,b 찾기

MSE + penalty를 최소로 만드는 w, b 찾기

MSE와 penalty를 최소가 되는 w와 b를 찾는 것이 핵심이다
이는 w의 모든 원소가 0이되거나 0에 가깝게 되야한다는 의미이다
그렇다면 어떤 특성들은 모델을 만들때 사용하지 않게 된다
모델을 만들때 모든 가중치를 고려하게 된다면 복잡한 모델이 만들어지고 그만큼 학습 데이터에만 잘 학습된
즉, overfitting이 된 모델이 탄생하게 될 것이다

따라서 penalty를 추가함으로써 자연스럽게 학습과정에서 덜 중요한 특징들은 사용되지 않게 적절한 특징을
추려내는 것이 관건이다
예를 들어 100개 특성을 갖는 회귀 모델일 있다고 가정할 때 α=1로 설정했더니 95개가 0이 되면서 5개의 특성이 사용된 것을 확인 했다. 이때 성능 측정을 했더니 underfitting이 발생했다
그런데 α=0.0001로 설정했더니 5개만 0이 되면서 95개 특성을 사용해야 했다. 이때 성능 측정을 했더니 overfitting이 발생했다
결국 α=0.1로 설정하여 70개 특성이 0이 되면서 30개 특성을 사용하여 성능 측정을 했더니 적절하게 일반화 된 모델을 만들 수 있게 되었다
이때 이 α를 구할 수 있도록 파라미터가 추가 된 것은 L1-norm penalty를 추가했기 때문에 가능한 일이다

Ridge

Ridge는 L1-norm 대신 L2-norm을 사용한 Regulaization 방법이라고 생각하면 된다
Lasso와 차이점이 있다면 가중치가 0에 가까워질 뿐 0이 되지는 않는다
왜냐 하면 Ridge는 절대값을 추가한 것이 아니라 제곱합이기 때문에 0이 되지 않는다

따라서 특성의 중요도가 전체적으로 비슷한 경우 Ridge가 좀 더 괜찮은 모델을 찾아줄 것이다

layer = tf.keras.layers.Dense(
    5, input_dim=5,
    kernel_initializer='ones',
    kernel_regularizer=tf.keras.regularizers.L1(0.01)
)

layer.weights
# []

layer.build((None,5))

layer.weights

[<tf.Variable 'kernel:0' shape=(5, 5) dtype=float32, numpy=
 array([[1., 1., 1., 1., 1.],
        [1., 1., 1., 1., 1.],
        [1., 1., 1., 1., 1.],
        [1., 1., 1., 1., 1.],
        [1., 1., 1., 1., 1.]], dtype=float32)>,
 <tf.Variable 'bias:0' shape=(5,) dtype=float32, numpy=array([0., 0., 0., 0., 0.], dtype=float32)>]

tensor = tf.ones(shape=(5,5))*2.0

tensor
<tf.Tensor: shape=(5, 5), dtype=float32, numpy=
array([[2., 2., 2., 2., 2.],
       [2., 2., 2., 2., 2.],
       [2., 2., 2., 2., 2.],
       [2., 2., 2., 2., 2.],
       [2., 2., 2., 2., 2.]], dtype=float32)>

out = layer(tensor)
out
<tf.Tensor: shape=(5, 5), dtype=float32, numpy=
array([[10., 10., 10., 10., 10.],
       [10., 10., 10., 10., 10.],
       [10., 10., 10., 10., 10.],
       [10., 10., 10., 10., 10.],
       [10., 10., 10., 10., 10.]], dtype=float32)>

Regularizer / penalty

각 항의 계수를 작게 만들어 가설함수를 간단하게 하고 과적합을 피할수 있도록 유도한다

loss = loss + regulizer_term(L1, L2)

(X_train, y_train), (X_test, y_test) = tf.keras.datasets.mnist.load_data()
y_train = tf.cast(y_train, tf.int32)

model = tf.keras.models.Sequential([
    tf.keras.layers.Flatten(input_shape=(28,28)),
    tf.keras.layers.Dense(128,activation='relu', kernel_regularizer=tf.keras.regularizers.l1()),
    tf.keras.layers.Dropout(0.2),
    tf.keras.layers.Dense(10, activation='softmax')
])

loss = tf.keras.losses.SparseCategoricalCrossentropy()

prediction = model(X_train)

model.losses # regularizer를 반환한다
# [<tf.Tensor: shape=(), dtype=float32, numpy=40.870396>]

# loss + Regularization term / overfitting은 덜 나지만 학습속도는 느려진다
loss(y_train, prediction) + tf.reduce_sum(model.losses) 
# <tf.Tensor: shape=(), dtype=float32, numpy=164.54881>

loss function의 차이

(X_train, y_train), (X_test, y_test) = tf.keras.datasets.mnist.load_data()
y_train = tf.cast(y_train, tf.int32)
y_train_ohe = tf.keras.utils.to_categorical(y_train)

model1 = tf.keras.models.Sequential([
    tf.keras.layers.Flatten(input_shape=(28,28)),
    tf.keras.layers.Dense(128,activation='relu', kernel_regularizer=tf.keras.regularizers.l1()),
    tf.keras.layers.Dropout(0.2),
    tf.keras.layers.Dense(10, activation='softmax')
])
prediction = model1(X_train)

tf.keras.losses.sparse_categorical_crossentropy(y_train, prediction)
# tf.keras.losses.SparseCategoricalCrossentropy()(y_train, prediction)
# tf.nn.sparse_softmax_cross_entropy_with_logits(y_train, prediction)

# <tf.Tensor: shape=(60000,), dtype=float32, numpy=
# array([192.52524 ,  51.914566,  72.927284, ..., 171.85602 , 150.09734 ,
#         53.307495], dtype=float32)>

# tf.keras.losses.sparse_categorical_crossentropy(y_train, prediction)
tf.keras.losses.SparseCategoricalCrossentropy()(y_train, prediction)
# tf.nn.sparse_softmax_cross_entropy_with_logits(y_train, prediction)

# <tf.Tensor: shape=(), dtype=float32, numpy=116.44715>

# tf.keras.losses.sparse_categorical_crossentropy(y_train, prediction)
# tf.keras.losses.SparseCategoricalCrossentropy()(y_train, prediction)
tf.nn.sparse_softmax_cross_entropy_with_logits(y_train, prediction)

# <tf.Tensor: shape=(60000,), dtype=float32, numpy=
# array([2.46115  , 2.46115  , 2.46115  , ..., 2.46115  , 2.46115  ,
#        2.4379332], dtype=float32)>

model2 = tf.keras.models.Sequential([
    tf.keras.layers.Flatten(input_shape=(28,28)),
    tf.keras.layers.Dense(128,activation='relu', kernel_regularizer=tf.keras.regularizers.l1()),
    tf.keras.layers.Dropout(0.2),
    tf.keras.layers.Dense(10)
])
prediction = model2(X_train)

tf.keras.losses.sparse_categorical_crossentropy(y_train, prediction)
# tf.keras.losses.SparseCategoricalCrossentropy(from_logits=True)(y_train, prediction)
# tf.nn.sparse_softmax_cross_entropy_with_logits(y_train, prediction)

# <tf.Tensor: shape=(60000,), dtype=float32, numpy=
# array([17.216707 , 17.727533 ,  1.6094378, ...,  1.0986124,  1.0986124,
#        17.216707 ], dtype=float32)>

# tf.keras.losses.sparse_categorical_crossentropy(y_train, prediction)
tf.keras.losses.SparseCategoricalCrossentropy(from_logits=True)(y_train, prediction)
# tf.nn.sparse_softmax_cross_entropy_with_logits(y_train, prediction)

# <tf.Tensor: shape=(), dtype=float32, numpy=151.73372>

# tf.keras.losses.sparse_categorical_crossentropy(y_train, prediction)
# tf.keras.losses.SparseCategoricalCrossentropy(from_logits=True)(y_train, prediction)
tf.nn.sparse_softmax_cross_entropy_with_logits(y_train, prediction)

# <tf.Tensor: shape=(60000,), dtype=float32, numpy=
# array([114.73    , 230.42961 , 114.96562 , ...,  13.309306,  -0.      ,
#        137.89368 ], dtype=float32)>

# model1.compile(loss=tf.keras.losses.sparse_categorical_crossentropy)
# model1.compile(loss=tf.keras.losses.SparseCategoricalCrossentropy())
model1.compile(loss=tf.nn.sparse_softmax_cross_entropy_with_logits)
# model1.compile(loss=tf.nn.softmax_cross_entropy_with_logits)

# model2.compile(loss=tf.keras.losses.sparse_categorical_crossentropy)
# model2.compile(loss=tf.keras.losses.SparseCategoricalCrossentropy())
model2.compile(loss=tf.nn.sparse_softmax_cross_entropy_with_logits)
# model2.compile(loss=tf.nn.softmax_cross_entropy_with_logits)

model1.fit(X_train, y_train, epochs=2) # sparse_categorical_crossentropy

Epoch 1/2
1875/1875 [==============================] - 6s 3ms/step - loss: 9.0639
Epoch 2/2
1875/1875 [==============================] - 6s 3ms/step - loss: 3.7299
<keras.callbacks.History at 0x2ab863250>

model1.fit(X_train, y_train, epochs=2) # SparseCategoricalCrossentropy

Epoch 1/2
32/1875 [..............................] - ETA: 6s - loss: 6.2577 
1875/1875 [==============================] - 6s 3ms/step - loss: 3.1716
Epoch 2/2
1875/1875 [==============================] - 6s 3ms/step - loss: 2.5536
<keras.callbacks.History at 0x2afab4bb0>

model1.fit(X_train, y_train_ohe, epochs=2) # nn.sparse_softmax_cross_entropy_with_logits

model1.fit(X_train, y_train, epochs=2) # nn.softmax_cross_entropy_with_logits

Epoch 1/2
  18/1875 [..............................] - ETA: 12s - loss: 97.9138 
  1875/1875 [==============================] - 9s 5ms/step - loss: 103.1872
Epoch 2/2
1875/1875 [==============================] - 9s 5ms/step - loss: 103.0049
<keras.callbacks.History at 0x2abb74610>

model2.fit(X_train, y_train_ohe, epochs=2) # nn.softmax_cross_entropy_with_logits (with softmax)

Epoch 1/2
  21/1875 [..............................] - ETA: 9s - loss: 91.7908
  1875/1875 [==============================] - 9s 5ms/step - loss: 8.9059
Epoch 2/2
1875/1875 [==============================] - 8s 4ms/step - loss: 3.7517
<keras.callbacks.History at 0x2ca7eba00>

model2.fit(X_train, y_train_ohe, epochs=2) # nn.sparse_softmax_cross_entropy_with_logits

One-hot encoding x => sparse O
One-hot encoding O => Sparse X

(X_train, y_train), (X_test, y_test) = tf.keras.datasets.mnist.load_data()

model = tf.keras.models.Sequential([
    tf.keras.layers.Conv2D(3,3),
    tf.keras.layers.Conv2D(1,3)
])

model(X_train, apply_sigmoid=True)

tensor = tf.ones(shape)

out = layer(tensor, apply_sigmoid=True)

저작자표시 비영리

'Computer_Science > Visual Intelligence' 카테고리의 다른 글

38일차 - Style Transfer (0)	2021.11.16
37일차 - 논문 수업 (Segmentation) (0)	2021.11.16
35일차 - 논문 수업 (Segmentation) (0)	2021.11.01
34일차 - 논문 수업 (VAE) (0)	2021.11.01
33일차 - 논문 수업 (VAE) (0)	2021.11.01

35일차 - 논문 수업 (Segmentation)

2021. 11. 1. 15:36

728x90

Upsampling

1. Nearest Neighbor

- N배 만큼 이미지를 키운다고 했을 때 Input 이미지에서 각 값에서 늘어난 부분에 가장 가까운 값을 채워 넣는다

2. Bed of Nails

- N배 만큼 이미지를 키운다고 했을 때 Input 이미지에서 늘어난 부분에 0값을 채워 넣는다

- ZFNet의 unpooling은 bed of nails의 일종이다

x = tf.reshape(tf.range(24), (1,2,3,4))
x
# <tf.Tensor: shape=(1, 2, 3, 4), dtype=int32, numpy=
# array([[[[ 0,  1,  2,  3],
#          [ 4,  5,  6,  7],
#          [ 8,  9, 10, 11]],
# 
#         [[12, 13, 14, 15],
#          [16, 17, 18, 19],
#          [20, 21, 22, 23]]]], dtype=int32)>

tf.keras.layers.UpSampling2D((1,2))(x)

# <tf.Tensor: shape=(1, 2, 6, 4), dtype=int32, numpy=
# array([[[[ 0,  1,  2,  3],
#          [ 0,  1,  2,  3],
#          [ 4,  5,  6,  7],
#          [ 4,  5,  6,  7],
#          [ 8,  9, 10, 11],
#          [ 8,  9, 10, 11]],
# 
#         [[12, 13, 14, 15],
#          [12, 13, 14, 15],
#          [16, 17, 18, 19],
#          [16, 17, 18, 19],
#          [20, 21, 22, 23],
 #         [20, 21, 22, 23]]]], dtype=int32)>

from scipy import ndimage, signal
x = tf.reshape(tf.range(9), (3,3))
f = tf.ones((2,2))
ndimage.convolve(x,f,mode='nearest') # 마지막 픽셀의 값을 복사하여 채워 넣고 연산한다 
# array([[ 8, 12, 14],
#        [20, 24, 26],
#        [26, 30, 32]], dtype=int32)

Convolution vs Transpose convolution vs Deconvolution

1. Convolution

- 이미지에서 특징을 추출하기 위한 연산 방법

- 보통 이미지에서 중요한 특징을 추출하고 연산량을 줄이기 위해 downsampling이 된다

- 데이터를 상하, 좌우 반전후 correlation연산을 하는 방법

2. Convolution transpose

- Learnable upsampling (보간법을 활용하여 값을 채우는 것과 비슷하다)

- convolution 연산과 전혀 다르지만 연산 결과가 같을 수 있다

- 보간법과 다르게 학습할 수 있는 파라미터가 있는 방법이다

3. Deconvolution

- convolution의 역연산

Transpose convolution

import tensorflow as tf 
import numpy as np

a = np.array([1,2,3])
b = np.array([-1,1])

signal.convolve(a,b) # convolution transpose 연산결과가 같다 

# array([-1, -1, -1,  3])

convolution transpose

1 -1 -1

2 -1 => -2 + 1 => -1

3 1 -3 + 2 -1

3 3

Variational AutoEncoder

내가 아는 확률 분포(정규분포)와 찾기 힘든분포를 가능한 근사적으로 하여 확률 분포를 찾아내는 방법

(사후확률 분포를 보다 단순한 확률분포(정규분포)로 근사하는 방법)

VAE 순서

1. 잠재변수는 다루기 쉬운 확률분포(다차원의 정규분포 등)로 가정한다

2. Encoder 함수를 통해서 평균과 표준편차를 추정한다

3. latent variable에서 sample된 z라는 value(decoder input)를 만든다 (Reparameterization trick)

4. z로부터 sampling을 하여 generator를 생성한다

Decoder network

ELBO(Evidence LowerBOund)

ELBO term을 Φ에 대해 maximize 하면 이상적인 sampling함수를 찾는 것이다
ELBO term을 𝜃에 대해 maximize 하면 MLE 관점에서 Network의 파라미터를 찾는 것이다
출처: https://deepinsight.tistory.com/127 [Steve-Lee's Deep Insight]

AutoEncoder vs Variational AutoEncoder

AutoEncoder와 Variational AutoEncoder 구조가 매우 유사한데

왜 AutoEncoder는 생성모델이 아니고 VAE는 생성 모델일까?

AutoEncoder는 사전 지식에 대한 조건이 없기 때문에 의미 있는 z vector의 latent space가 계속해서 바뀐다

즉 새로운 이미지를 생성할 때 z값이 계속해서 바뀐다

반면 VAE는 사전지식에 대한 조건을 부여 했기 때문에 z vector가 사전지식(내가 아는 분포)를 따른다

따라서 같은 분포에서 sampling을 하기 때문에 생성모델로 볼수 있다

저작자표시 비영리

'Computer_Science > Visual Intelligence' 카테고리의 다른 글

37일차 - 논문 수업 (Segmentation) (0)	2021.11.16
36일차 - 논문 수업 (Segmentation) (0)	2021.11.16
34일차 - 논문 수업 (VAE) (0)	2021.11.01
33일차 - 논문 수업 (VAE) (0)	2021.11.01
32일차 - 논문 수업 (AE) (0)	2021.11.01

34일차 - 논문 수업 (VAE)

2021. 11. 1. 15:30

728x90

AutoEncoder

import tensorflow as tf

(X_train, _), (X_test,_) = tf.keras.datasets.fashion_mnist.load_data()

X_train = X_train.reshape(60000, 28*28)
X_test = X_test.reshape(10000, 28*28)

X_train = X_train/255
X_test = X_test/255

class MyModel(tf.keras.models.Model):
  def __init__(self, dim):
    super(MyModel, self).__init__() # MyModel의 부모 객체를 불러와 init를 실행한다 
    self.dim = dim 
    self.encoder = tf.keras.layers.Dense(dim)
    self.decoder = tf.keras.layers.Dense(28*28, activation='sigmoid')
    # self.encoder = tf.keras.models.Sequential([ # 모델 안에 모델 넣는 방법 
    #     tf.keras.layers.Flatten(), # encoder 안에서 flatten 
    #     tf.keras.layers.Dense(dim, activation='relu')
    # ])
    # self.decoder = tf.keras.models.Sequential([
    #     tf.keras.layers.Dense(28*28, activation='sigmoid'),
    #     tf.keras.layers.Reshape(28,28)
    # ])

  def call(self, input_):
    x = self.encoder(input_)
    x = self.decoder(x)
    return x

autoencoder = MyModel(32)

autoencoder.compile(loss=tf.keras.losses.MeanSquaredError())

autoencoder.fit(X_train, X_train, epochs=20)
Epoch 1/20
1875/1875 [==============================] - 4s 2ms/step - loss: 0.0282
Epoch 2/20
1875/1875 [==============================] - 3s 2ms/step - loss: 0.0157
Epoch 3/20
1875/1875 [==============================] - 4s 2ms/step - loss: 0.0140
Epoch 4/20
1875/1875 [==============================] - 3s 2ms/step - loss: 0.0134
Epoch 5/20
1875/1875 [==============================] - 3s 2ms/step - loss: 0.0131
Epoch 6/20
1875/1875 [==============================] - 3s 2ms/step - loss: 0.0129
Epoch 7/20
1875/1875 [==============================] - 3s 2ms/step - loss: 0.0128
Epoch 8/20
1875/1875 [==============================] - 3s 2ms/step - loss: 0.0128
Epoch 9/20
1875/1875 [==============================] - 3s 2ms/step - loss: 0.0127
Epoch 10/20
1875/1875 [==============================] - 3s 2ms/step - loss: 0.0127
Epoch 11/20
1875/1875 [==============================] - 3s 2ms/step - loss: 0.0126
Epoch 12/20
1875/1875 [==============================] - 3s 2ms/step - loss: 0.0126
Epoch 13/20
1875/1875 [==============================] - 4s 2ms/step - loss: 0.0126
Epoch 14/20
1875/1875 [==============================] - 3s 2ms/step - loss: 0.0126
Epoch 15/20
1875/1875 [==============================] - 4s 2ms/step - loss: 0.0126
Epoch 16/20
1875/1875 [==============================] - 4s 2ms/step - loss: 0.0125
Epoch 17/20
1875/1875 [==============================] - 3s 2ms/step - loss: 0.0125
Epoch 18/20
1875/1875 [==============================] - 4s 2ms/step - loss: 0.0125
Epoch 19/20
1875/1875 [==============================] - 4s 2ms/step - loss: 0.0125
Epoch 20/20
1875/1875 [==============================] - 4s 2ms/step - loss: 0.0125
<keras.callbacks.History at 0x7fdfc80aab50>

autoencoder.encoder, autoencoder.decoder # 상속을 통해서 만들면 더 유연한 모델을 만들 수 있다 

# (<keras.engine.sequential.Sequential at 0x7fdfc49ca910>,
#  <keras.engine.sequential.Sequential at 0x7fdfc7feec90>)

Convolutional autoencoder

(X_train, _), (X_test,_) = tf.keras.datasets.fashion_mnist.load_data()

X_train[..., tf.newaxis].shape
# (60000, 28, 28, 1)

tf.expand_dims(X_train, axis=-1)
X_train = X_train.reshape(60000, 28,28,1)

X_test = X_test.reshape(10000,28,28,1)

input_ = tf.keras.Input((28,28,1))

x = tf.keras.layers.Conv2D(16,3, padding='same', activation='relu')(input_)
x = tf.keras.layers.MaxPooling2D(2,2, padding='same')(x)
x = tf.keras.layers.Conv2D(8,3, padding='same', activation='relu')(x)
x = tf.keras.layers.MaxPooling2D(2,2, padding='same')(x)
x = tf.keras.layers.Conv2D(8,3, padding='same', activation='relu')(x)
x = tf.keras.layers.MaxPooling2D(2,2, padding='same')(x)

x = tf.keras.layers.Conv2D(8,3, padding='same', activation='relu')(x)
x = tf.keras.layers.UpSampling2D((2,2))(x)
x = tf.keras.layers.Conv2D(8,3, padding='same', activation='relu')(x)
x = tf.keras.layers.UpSampling2D((2,2))(x)
x = tf.keras.layers.Conv2D(16,3, activation='relu')(x)
x = tf.keras.layers.UpSampling2D((2,2))(x)
# x = tf.keras.layers.Conv2D(16,3, padding='same', activation='relu')(x)

model = tf.keras.models.Model(input_,x)

model.summary() 
# 중간에 복원하는 과정이 맞지 않아도 학습을 통해서 최종결과에서 틀린 것을 보완된다 / Conv2DTranspose(deconvolution)을 사용하면 완전히 복원 가능하긴 하다 
Model: "model_6"
_________________________________________________________________
Layer (type)                 Output Shape              Param #   
=================================================================
input_1 (InputLayer)         [(None, 28, 28, 1)]       0         
_________________________________________________________________
conv2d_37 (Conv2D)           (None, 28, 28, 16)        160       
_________________________________________________________________
max_pooling2d_18 (MaxPooling (None, 14, 14, 16)        0         
_________________________________________________________________
conv2d_38 (Conv2D)           (None, 14, 14, 8)         1160      
_________________________________________________________________
max_pooling2d_19 (MaxPooling (None, 7, 7, 8)           0         
_________________________________________________________________
conv2d_39 (Conv2D)           (None, 7, 7, 8)           584       
_________________________________________________________________
max_pooling2d_20 (MaxPooling (None, 4, 4, 8)           0         
_________________________________________________________________
conv2d_40 (Conv2D)           (None, 4, 4, 8)           584       
_________________________________________________________________
up_sampling2d_16 (UpSampling (None, 8, 8, 8)           0         
_________________________________________________________________
conv2d_41 (Conv2D)           (None, 8, 8, 8)           584       
_________________________________________________________________
up_sampling2d_17 (UpSampling (None, 16, 16, 8)         0         
_________________________________________________________________
conv2d_42 (Conv2D)           (None, 14, 14, 16)        1168      
_________________________________________________________________
up_sampling2d_18 (UpSampling (None, 28, 28, 16)        0         
=================================================================
Total params: 4,240
Trainable params: 4,240
Non-trainable params: 0
_________________________________________________________________

Type Markdown and LaTeX: $α^{2}$

Variational AutoEncoder

숨어 있는 확률 분포가 복잡할 경우
데이터의 분포를 정규분포와 가능한 오차가 적은 방향으로 분포를 구한다

Likelihood

어떤 모델에서 해당 데이터(관측값)이 나올 확률

Maximum Likelihood

현재 가지고 있는 데이터셋이 나올 확률을 최대화하는 방향으로 확률 분포를 구하는 방법

데이터가 1, 4, 5, 6, 9가 있다고 가정했을 때

주황색 확률분포로 부터 데이터들을 얻었을 가능성이 더 클 것이다

왜냐하면 획득한 데이터들의 분포가 주황색 확률분포의 중심에 더 일치하는 것 처럼 보이기 때문이다

단, 주어진 데이터 샘플이 편향된 경우 추정된 확률 분포가 실제 상황을 표현하기에 부적합한 확률 분포가 될 수 있다

Maximum A Posterior

사전 지식에 근거하여(과거의 경험) 실제 데이터의 분포가 어떠 할 것이다 라고 예측하는 방법

동전 던지기를 예로 들면

동전을 10번 던져서 모두 앞면이 나온다고 하더라도,

동전의 앞면과 뒷면이 나올 확률이 0.5로 동일할거라는 사전지식이 동시에 작용하기 때문에

동전의 뒷면이 나올 확률이 0%가 되지 않도록 확률 분포를 구하는데 영향을 끼친다

따라서 올바른 사전지식을 가지는 것이 중요하며 올바른 사전지식을 활용하는 경우 MLE보다 더 나은 모수(평균, 표준편차등) 파라미터를 추정할 수 있다

생성 모델

숨어 있는 분포를 알 수 있으면 데이터를 생성할 수 있다

예를 들어 숫자의 잠재적인 의미 즉 필체를 나타내는 잠재변수인 각도, aspect ratio등으로 부터 숫자를 만들어 낼 수 있다

다차원의 아주 큰 분포를 더 작은 분포로 표현할 수 있는데

더 작은 분포를 찾으면 다차원의 데이터 분포를 가진 데이터를 생성 할 수 있다

작은 latent space를 통해 학습된 모델에서도 데이터를 잘 표현하는 데이터를 생성할 수 있다

※ 잠재 공간(latent space): 어떠한 관측 데이터 혹은 입력 데이터가 있을 때 그 데이터 들을 대표할 수 있는 함축된 데이터 공간 또는 집합을 잠재 공간이라고 한다

AutoEncoder의 생성 모델적 의미

Encoder를 이용해서 복잡한 분포로 부터 정규분포를 알아내고

정규 분포로부터 숨어 있는 작은 분포를 찾아낸다(latent space)

이렇게 알아낸 latent space로부터 decoder를 활용하여 원래의 데이터를 복원한다

왜도와 첨도

1. 왜도(skewness) - 분포의 비대칭성을 나타내는 척도

2. 첨도(kurtosis) - 분포의 뾰족한 정도를 나타내는 척도

왜도

첨도

import pandas as pd
import seaborn as sns
import matplotlib.pyplot as plt

iris = sns.load_dataset('iris')
data = iris.iloc[:,:-1]

f, axes = plt.subplots(2, 2, figsize=(7, 7), sharex=True)
sns.distplot(data.iloc[:,0], color="skyblue", ax=axes[0,0])
sns.distplot(data.iloc[:,1], color="olive", ax=axes[0,1])
sns.distplot(data.iloc[:,2], color="gold", ax=axes[1,0])
sns.distplot(data.iloc[:,3], color="teal", ax=axes[1,1])
for i, ax in enumerate(axes.reshape(-1)):
    ax.text(x=0.97, y=0.97, transform=ax.transAxes, s="Skewness: %f" % data.iloc[:,i].skew(),\
        fontweight='demibold', fontsize=10, verticalalignment='top', horizontalalignment='right',\
        backgroundcolor='white', color='xkcd:poo brown')
    ax.text(x=0.97, y=0.91, transform=ax.transAxes, s="Kurtosis: %f" % data.iloc[:,i].kurt(),\
        fontweight='demibold', fontsize=10, verticalalignment='top', horizontalalignment='right',\
        backgroundcolor='white', color='xkcd:dried blood')
plt.tight_layout()

저작자표시 비영리

'Computer_Science > Visual Intelligence' 카테고리의 다른 글

36일차 - 논문 수업 (Segmentation) (0)	2021.11.16
35일차 - 논문 수업 (Segmentation) (0)	2021.11.01
33일차 - 논문 수업 (VAE) (0)	2021.11.01
32일차 - 논문 수업 (AE) (0)	2021.11.01
31일차 - 논문 수업 (Detection) (0)	2021.11.01

33일차 - 논문 수업 (VAE)

2021. 11. 1. 15:25

728x90

YOLO (You Only Look Once)

import tensorflow as tf

vgg = tf.keras.applications.VGG16()

vgg.layers[2].get_config()

{'activation': 'relu',
 'activity_regularizer': None,
 'bias_constraint': None,
 'bias_initializer': {'class_name': 'Zeros', 'config': {}},
 'bias_regularizer': None,
 'data_format': 'channels_last',
 'dilation_rate': (1, 1),
 'dtype': 'float32',
 'filters': 64,
 'groups': 1,
 'kernel_constraint': None,
 'kernel_initializer': {'class_name': 'GlorotUniform',
  'config': {'seed': None}},
 'kernel_regularizer': None,
 'kernel_size': (3, 3),
 'name': 'block1_conv2',
 'padding': 'same',
 'strides': (1, 1),
 'trainable': True,
 'use_bias': True}

layer = tf.keras.layers.Dense(2)
layer.build((None,2))  

layer.weights
# [<tf.Variable 'kernel:0' shape=(2, 2) dtype=float32, numpy=
#  array([[-0.9223565 , -0.76073015],
#         [-0.94507563,  0.25733256]], dtype=float32)>,
#  <tf.Variable 'bias:0' shape=(2,) dtype=float32, numpy=array([0., 0.], dtype=float32)>]

layer2 = tf.keras.layers.Dense(2) # __call__는 build가 되어 있는지 확인하고 되어 있으면 call 연산  
layer2(tf.constant([[1,2,3],[4,5,6]])) # build가 되어 있지 않으면 build후 call 연산 

# <tf.Tensor: shape=(2, 2), dtype=float32, numpy=
# array([[ 2.7301223, -2.2875605],
#        [ 5.7739477, -3.6619558]], dtype=float32)>

model = tf.keras.models.Sequential([
    tf.keras.layers.Dense(2, input_shape(2,)) # (None,2) => 요소가 2개인 데이터 N개 
])

model(tf.constant([[1,2,3],[4,5,6]]))
# <tf.Tensor: shape=(2, 2), dtype=float32, numpy=
# array([[ 5.3184466,  3.66491  ],
#        [12.929317 ,  7.4551005]], dtype=float32)>

model.input_shape # build 하지 않으면 input_shape을 알 수 없다 
# (2, 3)

model.weights
# [<tf.Variable 'dense_3/kernel:0' shape=(3, 2) dtype=float32, numpy=
#  array([[ 0.6508478 , -0.3525918 ],
#         [ 0.99072933,  0.830464  ],
#         [ 0.89538   ,  0.7855246 ]], dtype=float32)>,
#  <tf.Variable 'dense_3/bias:0' shape=(2,) dtype=float32, numpy=array([0., 0.], dtype=float32)>]

Build의 의미

ex) layer = tf.keras.layer.Dense(2)

1. layer를 만들때 내부적으로 __init__가 실행되며 unit을 인자로 받는다

2. 구조적으로 인자로 받은 unit수 만큼 perceptron을 생성한다

3. input의 인자 개수에 따라 자동적으로 weight metrix의 크기가 결정된다

4. unit수 만큼 output으로 나온다

build는 weight metrix의 크기를 정하고 만드는 작업을 의미한다 + 초기화

Custom layer (YOLO)

class YoloReshape(tf.keras.layers.Layer):
  def __init__(self, shape_):
    super().__init__()
    self.shape_ = tuple(shape_)

  def call(self, input_):
    S = [self.shape_[0], self.shape_[1]]
    C = 20 
    B = 2
    S[0]*S[1]*C
    ix = S[0]*S[1]*C
    ix2 = S[0]*S[1]*B

    boxs = tf.reshape(input_[:,:ix2], tuple(input_.shape[0],S[0],S[1],B*4))
    boxs = tf.nn.sigmoid(boxs) 

    confidence = tf.reshape(input_[:,ix2:ix1], tuple(input_.shape[0],S[0],S[1],B))
    confidence = tf.nn.sigmoid(confidence) 

    class_prob = tf.reshape(input_[:,ix1:], tuple(input_.shape[0],S[0],S[1],C)) # input_.shape[0] + tuple(S[0],S[1],C) / 10,20
    class_prob = tf.nn.softmax(class_prob) # 확률로 강제시킨다 

    return tf.concat([boxs, confidence, class_prob])

  def build(self):
    pass

  def get_config(self):
    config = super().get_config().copy()
    config.update({
        'target_shape':self.shape_
    })
    return config

Representation learning

- 어떤 task를 수행하기에 적절하게 데이터의 representation을 변형하는 방법을 학습하는 것

- 어떤 task를 더 쉽게 수행할 수 있는 다른 형태로 표현을 만드는 것 예) PCA

AutoEncoder

Unsupervised learning
Self-supervised learning
Manifold learning (N차원의 공간을 N보다 작은 저차원의 특성으로 표현하여 학습하는 방법)
Feature extraction

Autoencoder는 내 자신을 작은 차원으로 표현하는 encoder와 작은 차원을 그대로 복원하는 decoder로 구성되어 있는 하나의 모델

Autoencoder를 이상치 제거에서 활용한다고 했을 때

Encoder를 거쳐 차원이 줄어들면 핵심 특징을 제외한 것들은 자연스럽게 사라지게 되므로

noise가 포함된 데이터가 autoencoder를 통과하면 noise는 사라지게 될 것이다

비용절감 차원에서 좋은 모델

self-supervised: 자기 자신의 데이터가 target이 된다

AutoEncoder 구현

import tensorflow as tf

(X_train, _), (X_test, _) = tf.keras.datasets.mnist.load_data()
X_train = X_train.reshape(60000,28*28)
X_train = X_train/255
X_test = X_test.reshape(10000,28*28)
X_test = X_test/255

input_ = tf.keras.Input((784,)) # 28x28 image이기 때문에 
encoder = tf.keras.layers.Dense(32)(input_) # layer는 encoder와 decoder가 대칭이 되도록 설계 해야 한다 
encoder = tf.keras.layers.ReLU()(encoder)
decoder = tf.keras.layers.Dense(784, activation='sigmoid')(encoder) # Mnist 

autoencoder = tf.keras.models.Model(input_, decoder)
autoencoder.compile(loss=tf.keras.losses.BinaryCrossentropy(), optimizer='adam')

autoencoder.summary()
Model: "model"
_________________________________________________________________
Layer (type)                 Output Shape              Param #   
=================================================================
input_1 (InputLayer)         [(None, 784)]             0         
_________________________________________________________________
dense (Dense)                (None, 32)                25120     
_________________________________________________________________
re_lu (ReLU)                 (None, 32)                0         
_________________________________________________________________
dense_1 (Dense)              (None, 784)               25872     
=================================================================
Total params: 50,992
Trainable params: 50,992
Non-trainable params: 0
_________________________________________________________________

history = autoencoder.fit(X_train, X_train, epochs=50)

Epoch 1/50
1875/1875 [==============================] - 5s 2ms/step - loss: 0.1629
Epoch 2/50
1875/1875 [==============================] - 4s 2ms/step - loss: 0.1044
Epoch 3/50
1875/1875 [==============================] - 4s 2ms/step - loss: 0.0966
Epoch 4/50
1875/1875 [==============================] - 4s 2ms/step - loss: 0.0951
Epoch 5/50
1875/1875 [==============================] - 4s 2ms/step - loss: 0.0946
Epoch 6/50
1875/1875 [==============================] - 4s 2ms/step - loss: 0.0942
Epoch 7/50
1875/1875 [==============================] - 4s 2ms/step - loss: 0.0940
Epoch 8/50
1875/1875 [==============================] - 4s 2ms/step - loss: 0.0939
Epoch 9/50
1875/1875 [==============================] - 4s 2ms/step - loss: 0.0938
Epoch 10/50
1875/1875 [==============================] - 4s 2ms/step - loss: 0.0937
Epoch 11/50
1875/1875 [==============================] - 4s 2ms/step - loss: 0.0936
Epoch 12/50
1875/1875 [==============================] - 4s 2ms/step - loss: 0.0935
Epoch 13/50
1875/1875 [==============================] - 4s 2ms/step - loss: 0.0935
Epoch 14/50
1875/1875 [==============================] - 4s 2ms/step - loss: 0.0934
Epoch 15/50
1875/1875 [==============================] - 4s 2ms/step - loss: 0.0934
Epoch 16/50
1875/1875 [==============================] - 4s 2ms/step - loss: 0.0934
Epoch 17/50
1875/1875 [==============================] - 4s 2ms/step - loss: 0.0933
Epoch 18/50
1875/1875 [==============================] - 4s 2ms/step - loss: 0.0933
Epoch 19/50
1875/1875 [==============================] - 4s 2ms/step - loss: 0.0932
Epoch 20/50
1875/1875 [==============================] - 4s 2ms/step - loss: 0.0932
Epoch 21/50
1875/1875 [==============================] - 4s 2ms/step - loss: 0.0932
Epoch 22/50
1875/1875 [==============================] - 4s 2ms/step - loss: 0.0931
Epoch 23/50
1875/1875 [==============================] - 4s 2ms/step - loss: 0.0931
Epoch 24/50
1875/1875 [==============================] - 4s 2ms/step - loss: 0.0931
Epoch 25/50
1875/1875 [==============================] - 4s 2ms/step - loss: 0.0931
Epoch 26/50
1875/1875 [==============================] - 4s 2ms/step - loss: 0.0930
Epoch 27/50
1875/1875 [==============================] - 4s 2ms/step - loss: 0.0930
Epoch 28/50
1875/1875 [==============================] - 4s 2ms/step - loss: 0.0930
Epoch 29/50
1875/1875 [==============================] - 4s 2ms/step - loss: 0.0930
Epoch 30/50
1875/1875 [==============================] - 4s 2ms/step - loss: 0.0929
Epoch 31/50
1875/1875 [==============================] - 4s 2ms/step - loss: 0.0929
Epoch 32/50
1875/1875 [==============================] - 4s 2ms/step - loss: 0.0929
Epoch 33/50
1875/1875 [==============================] - 4s 2ms/step - loss: 0.0929
Epoch 34/50
1875/1875 [==============================] - 4s 2ms/step - loss: 0.0929
Epoch 35/50
1875/1875 [==============================] - 4s 2ms/step - loss: 0.0928
Epoch 36/50
1875/1875 [==============================] - 4s 2ms/step - loss: 0.0928
Epoch 37/50
1875/1875 [==============================] - 4s 2ms/step - loss: 0.0928
Epoch 38/50
1875/1875 [==============================] - 4s 2ms/step - loss: 0.0928
Epoch 39/50
1875/1875 [==============================] - 4s 2ms/step - loss: 0.0928
Epoch 40/50
1875/1875 [==============================] - 4s 2ms/step - loss: 0.0927
Epoch 41/50
1875/1875 [==============================] - 4s 2ms/step - loss: 0.0928
Epoch 42/50
1875/1875 [==============================] - 4s 2ms/step - loss: 0.0927
Epoch 43/50
1875/1875 [==============================] - 4s 2ms/step - loss: 0.0927
Epoch 44/50
1875/1875 [==============================] - 4s 2ms/step - loss: 0.0927
Epoch 45/50
1875/1875 [==============================] - 4s 2ms/step - loss: 0.0927
Epoch 46/50
1875/1875 [==============================] - 4s 2ms/step - loss: 0.0927
Epoch 47/50
1875/1875 [==============================] - 4s 2ms/step - loss: 0.0926
Epoch 48/50
1875/1875 [==============================] - 4s 2ms/step - loss: 0.0926
Epoch 49/50
1875/1875 [==============================] - 4s 2ms/step - loss: 0.0926
Epoch 50/50
1875/1875 [==============================] - 4s 2ms/step - loss: 0.0926

import matplotlib.pyplot as plt

img = 7

fig, axs = plt.subplots(1,2)
axs[0].imshow(X_test[img].reshape(28,28))
axs[1].imshow(autoencoder(X_test)[img].numpy().reshape(28,28)) 
# 대표적인 특징만 표현된다 (섬세하게 재현되지는 않는다)

layer = tf.keras.layers.Dense(1)

layer.get_config() 
# activity_regularizer:  레이어 출력에 페널티를 적용하는 정규화 기법 / BN 때문에 더 이상 잘 사용하지 않는다
{'activation': 'linear',
 'activity_regularizer': None,
 'bias_constraint': None,
 'bias_initializer': {'class_name': 'Zeros', 'config': {}},
 'bias_regularizer': None,
 'dtype': 'float32',
 'kernel_constraint': None,
 'kernel_initializer': {'class_name': 'GlorotUniform',
  'config': {'seed': None}},
 'kernel_regularizer': None,
 'name': 'dense_2',
 'trainable': True,
 'units': 1,
 'use_bias': True}

Convolution 결과 복원

input_ = tf.keras.Input((28,28,1))
x = tf.keras.layers.Conv2D(16,3, padding='same')(input_)
x = tf.keras.layers.Conv2D(8,3, padding='same')(x)
x = tf.keras.layers.Conv2D(16,3, padding='same')(x)

model = tf.keras.models.Model(input_, x)

model.summary()
Model: "model_2"
_________________________________________________________________
Layer (type)                 Output Shape              Param #   
=================================================================
input_4 (InputLayer)         [(None, 28, 28, 1)]       0         
_________________________________________________________________
conv2d_6 (Conv2D)            (None, 28, 28, 16)        160       
_________________________________________________________________
conv2d_7 (Conv2D)            (None, 28, 28, 8)         1160      
_________________________________________________________________
conv2d_8 (Conv2D)            (None, 28, 28, 16)        1168      
=================================================================
Total params: 2,488
Trainable params: 2,488
Non-trainable params: 0
_________________________________________________________________

Sparse Auto-Encoder

차원을 늘리는 모델

저작자표시 비영리

'Computer_Science > Visual Intelligence' 카테고리의 다른 글

35일차 - 논문 수업 (Segmentation) (0)	2021.11.01
34일차 - 논문 수업 (VAE) (0)	2021.11.01
32일차 - 논문 수업 (AE) (0)	2021.11.01
31일차 - 논문 수업 (Detection) (0)	2021.11.01
30일차 논문 수업 (Detection) (0)	2021.11.01

32일차 - 논문 수업 (AE)

2021. 11. 1. 15:18

728x90

YOLO (You Only Look Once)

XML 파일 불러오기

Multiple object detection인 경우 정형데이터로 구성할 수 없기 때문에

xml이나 json과 같은 다양한 의미를 지닌 데이터 형식을 불러와야 할 때가 있다

import tensorflow as tf 
import xml.etree.ElementTree as ET # xml 형식의 파일을 parsing하는 파이썬 라이브러리 
import pathlib
import matplotlib.pyplot as plt
import matplotlib.patches as mpt

src = 'VOCdevkit/VOC2007/Annotations/000005.xml'

출처: http://host.robots.ox.ac.uk/pascal/VOC/voc2007/examples/index.html

tree = ET.parse(src) # xml 파알 구조 불러오기 (tree 형태로)
root = tree.getroot() # 시작지점 설정 (root 찾기)

root # annotation가 최상위 root (태그)
# <Element 'annotation' at 0x16107dcc0>

for i in root.iter('object'): # 5개 object가 있다 / name / pose / truncated /diffiicult / bndbox
    print(i)
# <Element 'object' at 0x163992a90>
# <Element 'object' at 0x163992db0>
# <Element 'object' at 0x163995130>
# <Element 'object' at 0x163995450>
# <Element 'object' at 0x163995770>

for i in root.iter('object'): # object에 대한 정보만 필요하기 때문에 object태그의 하위 요소만 불러온다
    difficult = i.find('difficult').text
    cls = i.find('name').text

Class 이름 key, value 쌍 만들기

classes = '''
Aeroplanes
Bicycles
Birds
Boats
Bottles
Buses
Cars
Cats
Chairs
Cows
Dining tables
Dogs
Horses
Motorbikes
People
Potted plants
Sheep
Sofas
Trains
TVMonitors
'''

classes = [i.lower() for i in classes.strip().split('\n')]

classes = [i[:-1] if i[-1]=='s' else i for i in classes]

classes
['aeroplane',
 'bicycle',
 'bird',
 'boat',
 'bottle',
 'buse',
 'car',
 'cat',
 'chair',
 'cow',
 'dining table',
 'dog',
 'horse',
 'motorbike',
 'people',
 'potted plant',
 'sheep',
 'sofa',
 'train',
 'tvmonitor']

classes = ['aeroplane',
 'bicycle',
 'bird',
 'boat',
 'bottle',
 'bus',
 'car',
 'cat',
 'chair',
 'cow',
 'diningtable',
 'dog',
 'horse',
 'motorbike',
 'people',
 'pottedplant',
 'sheep',
 'sofa',
 'train',
 'tvmonitor']

classes = dict(zip(classes, range(20)))

def annote(img_id):
    '''
    20개 클래스에 해당하는 class id와 4가지 위치 정보를 반환하는 함수 
    '''
    src = f'VOCdevkit/VOC2007/Annotations/{img_id}.xml'
    tree = ET.parse(src)
    root = tree.getroot()
    temp = []
    for i in root.iter('object'):
        difficult = i.find('difficult').text
        cls = i.find('name').text
        if cls not in classes: #  or int(difficult) == 1
            continue
        cls_id = classes[cls]
        xmlbox = i.find('bndbox')
        xmin = int(xmlbox.find('xmin').text)
        ymin = int(xmlbox.find('ymin').text)
        xmax = int(xmlbox.find('xmax').text)
        ymax = int(xmlbox.find('ymax').text)
        
        bb = (xmin,ymin,xmax,ymax)
        temp.append((cls_id, bb))
    return temp

path = pathlib.Path('VOCdevkit/VOC2007/Annotations')

for i in path.iterdir():
    print(str(i).split('/')[-1].split('.')[0])
007826
002786
006286
002962
...
004488
005796
002947

imgs = [str(i).split('/')[-1].split('.')[0] for i in path.iterdir()]
imgs
['007826',
 '002786',
 '006286',
 '002962',
 '008297',
 ...
 '004682',
 '006095',
 '001922',
 '000382',
 ...]

'VOCdevkit/VOC2007/JPEGImages/'+imgs[0]+'.jpg'
# 'VOCdevkit/VOC2007/JPEGImages/007826.jpg'

temp = [
]
for i in imgs:
    temp.append(annote(i))

len(temp)
# 5011

temp

[[(10, (80, 217, 320, 273)),
  (8, (197, 193, 257, 326)),
  (8, (139, 184, 185, 231)),
  ...
  [(9, (264, 148, 496, 280)), (12, (98, 123, 237, 248))],
 [(0, (65, 75, 470, 269))],
 [(3, (3, 167, 163, 500))],
 ...]

annote(imgs[0])
[(10, (80, 217, 320, 273)),
 (8, (197, 193, 257, 326)),
 (8, (139, 184, 185, 231)),
 (8, (258, 180, 312, 314)),
 (8, (10, 195, 93, 358)),
 (8, (82, 252, 243, 372)),
 (8, (43, 319, 144, 375))]

def show(img_id):
    '''
    이미지에 bounding box 그리는 함수 
    '''
    im = plt.imread('VOCdevkit/VOC2007/JPEGImages/'+str(imgs[img_id])+'.jpg')
    fig, ax = plt.subplots(1,1)
    for i in annote(imgs[img_id]):
        rec = mpt.Rectangle((i[1][0],i[1][1]), i[1][2]-i[1][0], i[1][3]-i[1][1], edgecolor='red', fill=False)
        ax.add_patch(rec)
    ax.imshow(im)

show(100)

show(1100)

데이터 불러와서 학습 데이터 구성하는 방법 (모델에 학습 데이터를 전달하기 위한/Multiple object)

1. 데이터를 파일로 부터 불러온다

2. 원하는 데이터 구조로 데이터를 가공한다 # ex) 'filename', (10,20,100,200), '1'

3. DataFrame 형태로 변환한다

4. Generator 형태로 만들거나 tf.data.Dataset으로 만든다

# 상속하는 방식(프레임워크에 한정적이지만 tensorflow에 최적화된 기능을 사용할 수 있다) / seqence data처럼 사용할 수 있기 때문에 학습 데이터 유연하게 전달할 수 있다
tf.keras.utils.Sequence 
# keras.utils.data_utils.Sequence

tf.keras.preprocessing.image_dataset_from_directory()
tf.keras.preprocessing.image.ImageDataGenerator() 
flow_from_directory 
flow_from_dataframe  # geneartor tf.data.Dataset

Numpy 형태 데이터를 학습을 하면 자동으로 tensor로 변환하여 학습이 가능하긴 하지만

이러한 tensor는 데이터 전체 메모리를 한꺼번에 올리기 때문에 컴퓨터 성능에 따라 학습이 불가능할 수도 있다

뿐만아니라 메모리를 효율적으로 사용하지 못하기 때문에 학습 속도가 비교적 느릴 수 있다

반면 tf.data.Dataset은 내부적으로 최적화 되어 있기 때문에 cpu, gpu를 동시에 사용하거나

prefetch, cache기법 등 다양한 방법으로 최적화된 학습이 가능하다

따라서 데이터를 generator형태로 불러오거나 tf.data.Dataset으로 변환하는 과정은 중요하다

(단, lazy 방식으로 만들어져 있기 때문에 데이터 하나 확인하는 것이 매우 까다롭다

Network Design

lre = tf.keras.layers.LeakyReLU(alpha=0.1) # 자주 쓰는 것들은 layers에 포함되어 있다 

input_ = tf.keras.Input((448,448,3))
x = tf.keras.layers.Conv2D(64,7,2, padding='same', activation=lre)(input_) 
x = tf.keras.layers.MaxPool2D(2,2)(x)
x = tf.keras.layers.Conv2D(192,3, padding='same', activation=lre)(x)
x = tf.keras.layers.MaxPool2D(2,2)(x)
x = tf.keras.layers.Conv2D(128,1, padding='same', activation=lre)(x)
x = tf.keras.layers.Conv2D(256,3, padding='same', activation=lre)(x)
x = tf.keras.layers.Conv2D(256,1, padding='same', activation=lre)(x)
x = tf.keras.layers.Conv2D(512,3, padding='same', activation=lre)(x)
x = tf.keras.layers.MaxPool2D(2,2)(x)
x = tf.keras.layers.Conv2D(256,1, padding='same', activation=lre)(x)
x = tf.keras.layers.Conv2D(512,3, padding='same', activation=lre)(x)
x = tf.keras.layers.Conv2D(256,1, padding='same', activation=lre)(x)
x = tf.keras.layers.Conv2D(512,3, padding='same', activation=lre)(x)
x = tf.keras.layers.Conv2D(256,1, padding='same', activation=lre)(x)
x = tf.keras.layers.Conv2D(512,3, padding='same', activation=lre)(x)
x = tf.keras.layers.Conv2D(256,1, padding='same', activation=lre)(x)
x = tf.keras.layers.Conv2D(512,3, padding='same', activation=lre)(x)
x = tf.keras.layers.Conv2D(512,1, padding='same', activation=lre)(x)
x = tf.keras.layers.Conv2D(1024,3, padding='same', activation=lre)(x)
x = tf.keras.layers.MaxPool2D(2,2)(x)
x = tf.keras.layers.Conv2D(512,1, padding='same', activation=lre)(x)
x = tf.keras.layers.Conv2D(1024,3, padding='same', activation=lre)(x)
x = tf.keras.layers.Conv2D(512,1, padding='same', activation=lre)(x)
x = tf.keras.layers.Conv2D(1024,3, padding='same', activation=lre)(x)
x = tf.keras.layers.Conv2D(1024,3,2, padding='same', activation=lre)(x)
x = tf.keras.layers.Conv2D(1024,3, padding='same', activation=lre)(x)
x = tf.keras.layers.Conv2D(1024,3, padding='same', activation=tf.keras.activations.linear)(x)

x = tf.keras.layers.Flatten()(x)
x = tf.keras.layers.Dense(4096, activation=lre)(x)
x = tf.keras.layers.Dropout(0.5)(x)
x = tf.keras.layers.Dense(1470)(x) 
x = tf.keras.layers.Reshape((7,7,30))(x)

model = tf.keras.models.Model(input_, x)

model.summary()

Model: "model"
_________________________________________________________________
Layer (type)                 Output Shape              Param #   
=================================================================
input_1 (InputLayer)         [(None, 448, 448, 3)]     0         
_________________________________________________________________
conv2d (Conv2D)              (None, 224, 224, 64)      9472      
_________________________________________________________________
max_pooling2d (MaxPooling2D) (None, 112, 112, 64)      0         
_________________________________________________________________
conv2d_1 (Conv2D)            (None, 112, 112, 192)     110784    
_________________________________________________________________
max_pooling2d_1 (MaxPooling2 (None, 56, 56, 192)       0         
_________________________________________________________________
conv2d_2 (Conv2D)            (None, 56, 56, 128)       24704     
_________________________________________________________________
conv2d_3 (Conv2D)            (None, 56, 56, 256)       295168    
_________________________________________________________________
conv2d_4 (Conv2D)            (None, 56, 56, 256)       65792     
_________________________________________________________________
conv2d_5 (Conv2D)            (None, 56, 56, 512)       1180160   
_________________________________________________________________
max_pooling2d_2 (MaxPooling2 (None, 28, 28, 512)       0         
_________________________________________________________________
conv2d_6 (Conv2D)            (None, 28, 28, 256)       131328    
_________________________________________________________________
conv2d_7 (Conv2D)            (None, 28, 28, 512)       1180160   
_________________________________________________________________
conv2d_8 (Conv2D)            (None, 28, 28, 256)       131328    
_________________________________________________________________
conv2d_9 (Conv2D)            (None, 28, 28, 512)       1180160   
_________________________________________________________________
conv2d_10 (Conv2D)           (None, 28, 28, 256)       131328    
_________________________________________________________________
conv2d_11 (Conv2D)           (None, 28, 28, 512)       1180160   
_________________________________________________________________
conv2d_12 (Conv2D)           (None, 28, 28, 256)       131328    
_________________________________________________________________
conv2d_13 (Conv2D)           (None, 28, 28, 512)       1180160   
_________________________________________________________________
conv2d_14 (Conv2D)           (None, 28, 28, 512)       262656    
_________________________________________________________________
conv2d_15 (Conv2D)           (None, 28, 28, 1024)      4719616   
_________________________________________________________________
max_pooling2d_3 (MaxPooling2 (None, 14, 14, 1024)      0         
_________________________________________________________________
conv2d_16 (Conv2D)           (None, 14, 14, 512)       524800    
_________________________________________________________________
conv2d_17 (Conv2D)           (None, 14, 14, 1024)      4719616   
_________________________________________________________________
conv2d_18 (Conv2D)           (None, 14, 14, 512)       524800    
_________________________________________________________________
conv2d_19 (Conv2D)           (None, 14, 14, 1024)      4719616   
_________________________________________________________________
conv2d_20 (Conv2D)           (None, 7, 7, 1024)        9438208   
_________________________________________________________________
conv2d_21 (Conv2D)           (None, 7, 7, 1024)        9438208   
_________________________________________________________________
conv2d_22 (Conv2D)           (None, 7, 7, 1024)        9438208   
_________________________________________________________________
flatten (Flatten)            (None, 50176)             0         
_________________________________________________________________
dense (Dense)                (None, 4096)              205524992 
_________________________________________________________________
dropout (Dropout)            (None, 4096)              0         
_________________________________________________________________
dense_1 (Dense)              (None, 1470)              6022590   
_________________________________________________________________
reshape (Reshape)            (None, 7, 7, 30)          0         
=================================================================
Total params: 262,265,342
Trainable params: 262,265,342
Non-trainable params: 0
_________________________________________________________________

Leaky ReLU

Loss function

10개 -> 두 개의 bounding box / 20개 -> 각각의 클래스 확률 결과를 도출하기 위해서는

Custom layers를 통해 직접 loss function을 구현해야 한다

Regression

Activation function을 linear함수(값의 범위만 정한다)를 사용하거나 사용하지 않으면 Regression 모델을 만들 수 있다

Loss function을 Crossentropy대신 MSE를 사용하면 된다

!pip install -U tensorflow-addons
# 최신 기법 + 중요성이 살짝 떨어지는 것들을 모아둔 추가 패키지 / 기본 tensorflow에 없는 layer를 사용할 수 있다

!pip install typeguard

# How to install tensorflow addons on mac with m1
!pip install --upgrade --force --no-dependencies https://github.com/apple/tensorflow_macos/releases/download/v0.1alpha3/tensorflow_addons_macos-0.1a3-cp38-cp38-macosx_11_0_arm64.whl

저작자표시 비영리

'Computer_Science > Visual Intelligence' 카테고리의 다른 글

34일차 - 논문 수업 (VAE) (0)	2021.11.01
33일차 - 논문 수업 (VAE) (0)	2021.11.01
31일차 - 논문 수업 (Detection) (0)	2021.11.01
30일차 논문 수업 (Detection) (0)	2021.11.01
30일차 - 논문 수업 (Detection) (0)	2021.10.24

31일차 - 논문 수업 (Detection)

2021. 11. 1. 15:10

728x90

import matplotlib.pyplot as plt
import matplotlib.patches as mpt
import pandas as pd
import tensorflow as tf
import numpy as np
import cv2

air = pd.read_csv('dataset/annotations/airplane.csv', header=None)
face = pd.read_csv('dataset/annotations/face.csv', header=None)
motorcycle = pd.read_csv('dataset/annotations/motorcycle.csv', header=None)

air.rename(columns={1:'x1',2:'y1',3:'x2',4:'y2',0:'filename',5:'target'}, inplace=True)
face.rename(columns={1:'x1',2:'y1',3:'x2',4:'y2',0:'filename',5:'target'}, inplace=True)
motorcycle.rename(columns={1:'x1',2:'y1',3:'x2',4:'y2',0:'filename',5:'target'}, inplace=True)

air.filename = air.filename.map(lambda x: 'dataset/images/airplane/'+x)
face.filename = face.filename.map(lambda x: 'dataset/images/face/'+x)
motorcycle.filename = motorcycle.filename.map(lambda x: 'dataset/images/motorcycle/'+x)

data=pd.concat([air,face,motorcycle], axis=0, ignore_index=True)

im = plt.imread('dataset/images/airplane/image_0001.jpg')

fig, ax = plt.subplots(1,1)
ax.imshow(im)
pt = mpt.Rectangle((49,30),349-49,137-30, fill=False) 
ax.add_patch(pt)

vgg = tf.keras.applications.VGG16(include_top=False, input_shape=(224,224,3))
vgg.trainable = False

vgg.summary()
Model: "vgg16"
_________________________________________________________________
Layer (type)                 Output Shape              Param #   
=================================================================
input_1 (InputLayer)         [(None, 224, 224, 3)]     0         
_________________________________________________________________
block1_conv1 (Conv2D)        (None, 224, 224, 64)      1792      
_________________________________________________________________
block1_conv2 (Conv2D)        (None, 224, 224, 64)      36928     
_________________________________________________________________
block1_pool (MaxPooling2D)   (None, 112, 112, 64)      0         
_________________________________________________________________
block2_conv1 (Conv2D)        (None, 112, 112, 128)     73856     
_________________________________________________________________
block2_conv2 (Conv2D)        (None, 112, 112, 128)     147584    
_________________________________________________________________
block2_pool (MaxPooling2D)   (None, 56, 56, 128)       0         
_________________________________________________________________
block3_conv1 (Conv2D)        (None, 56, 56, 256)       295168    
_________________________________________________________________
block3_conv2 (Conv2D)        (None, 56, 56, 256)       590080    
_________________________________________________________________
block3_conv3 (Conv2D)        (None, 56, 56, 256)       590080    
_________________________________________________________________
block3_pool (MaxPooling2D)   (None, 28, 28, 256)       0         
_________________________________________________________________
block4_conv1 (Conv2D)        (None, 28, 28, 512)       1180160   
_________________________________________________________________
block4_conv2 (Conv2D)        (None, 28, 28, 512)       2359808   
_________________________________________________________________
block4_conv3 (Conv2D)        (None, 28, 28, 512)       2359808   
_________________________________________________________________
block4_pool (MaxPooling2D)   (None, 14, 14, 512)       0         
_________________________________________________________________
block5_conv1 (Conv2D)        (None, 14, 14, 512)       2359808   
_________________________________________________________________
block5_conv2 (Conv2D)        (None, 14, 14, 512)       2359808   
_________________________________________________________________
block5_conv3 (Conv2D)        (None, 14, 14, 512)       2359808   
_________________________________________________________________
block5_pool (MaxPooling2D)   (None, 7, 7, 512)         0         
=================================================================
Total params: 14,714,688
Trainable params: 0
Non-trainable params: 14,714,688
_________________________________________________________________

im = tf.keras.preprocessing.image.load_img('dataset/images/airplane/image_0001.jpg')
t = cv2.resize(np.array(im), (224,224))
t.shape
# (224, 224, 3)

tt = vgg(t[tf.newaxis]).numpy()

tt.shape
# (1, 7, 7, 512)

plt.imshow(tt[0,...,0]) 
# 비행기 이미지를 convolution layer를 통과 했을 때 / 512개 특징중 비행기 특징 중 하나

plt.imshow(tt[0,...,1])

Yolo(You Only Look Once)

YOLO Detection System

1. ImageNet에서 사용했던 이미지 크기에서 두 배 키운 크기인 448x448 이미지로 크기를 변경한다

(Convolution 크기 mapping 하기 위해서)

2. Single convolutional neural network를 통과 시킨다

3. Non-max suppression

YOLO Architecture

1. Pretrained network

- 주황색 테두리로 표현한 부분은 GoogLeNet을 이용하여 ImageNet 1000-class dataset을 사전에 학습한 결과를 Fine-Tuning한 네트워크이다

- ImageNet의 데이터 셋은 224x224의 크기를 가진 이미지 데이터이지만, 이미지를 두 배 키워 448 x 448 x 3의 이미지를 입력 데이터로 받는다

2. Reduction layer

- Reduction layer는 GoogLeNet의 기법을 응용 하여 연산량은 감소하면서 층은 깊게 쌓는 방식을 이용한다

3. Training network

- Pre-trained Network에서 학습한 feature를 이용하여 Class probability와 Bounding box를 학습하고 예측하는 네트워크 이다

- YOLO의 예측 모델은 S x S x (B x 5 + C) 개의 파라미터를 결과로 출력하는데, PASCAL VOC 데이터 셋은 20개 클래스를 제공하기 때문에 C의 값은 20이고 B의 값이 2이라는 것은 하나의 그리드 셀에서 2개의 bounding box를 예측하겠다는 의미 이다.

그리고 그리드 셀은 7개로 정의 하였으므로 모두 계산하면 총 결과값은 1470이다 (7 x 7 x (2 X 5 + 20)

NMS(Non Maximum Suppression)

edge thinning 기법으로, 여러 box가 겹치게 되면 가장 확실한 것만 고르는 방법

1. Confidence(score)가 높은 순서대로 정렬한다

2. 제일 큰 Confidence값을 기준으로 하나씩 IOU를 비교 하여, 미리 설정해둔 threshold값보다 큰 경우 Bounding box를 버린다

3. 그 다음 작은 Confidence값을 가진 Bounding box를 기준으로 다시 자신보다 작은 confidence를 가진 Bounding box와 IOU를 비교하여 Bounding box를 선택할지 버릴지 결정한다

4. 3번 작업을 지속하여 최종적으로 남은 Bounding box를 사용한다

import tensorflow as tf 
import matplotlib.pyplot as plt
import matplotlib.patches as mpt

boxes = tf.constant([
    [190,380,(190+300),(380+150)],
    [300,420,(300+150),(420+210)],
    [320,360,(320+200),(360+200)],
    [390,50,(390+300),(50+300)],
    [390,50,(390+300),(50+300)],
], dtype=tf.float32)

scores = tf.constant([0.9,0.8,0.5,0.7,0.7])

tf.image.non_max_suppression(boxes, scores, 2)
# <tf.Tensor: shape=(2,), dtype=int32, numpy=array([0, 1], dtype=int32)>

fig, ax = plt.subplots(1,1)
im = plt.imread('people.jpg')
plt.imshow(im)
m1 = mpt.Rectangle((60,80),130,200, edgecolor='red', facecolor='none', linewidth=3)
m2 = mpt.Rectangle((90,60),110,50, edgecolor='pink', facecolor='none', linewidth=3)
ax.add_patch(m1)
ax.add_patch(m2)

fig, ax = plt.subplots(1,1)
m1 = mpt.Rectangle((60,80),130,200, edgecolor='red', facecolor='none', linewidth=5)
m2 = mpt.Rectangle((90,60),110,50, edgecolor='pink', facecolor='none', linewidth=5)
ax.add_patch(m1)
ax.add_patch(m2)
plt.plot();

NMS 처리 후 최소한의 Bounding box만 남긴다

Loss function

- 1i^obj는 그리드 셀 i 안에 객체가 존재하는지 여부를 의미한다. 이 값은 객체 존재하면 1, 존재하지 않으면 0이다

- 1ij^obj는 그리드 셀 i의 j번째 bounding box predictor가 사용되는지 여부를 의미한다.

- Object가 존재하는 그리드 셀 i의 bounding box predictor j에 대해, x와 y의 loss를 계산한다

- Object가 존재하는 그리드 셀 i의 bounding box predictor j에 대해, w와 h의 loss를 계산한다. 큰 box에 대해서는 작은 분산(small deviation)을 반영하기 위해 제곱근을 취한 후, sum-squared error를 구한다. (같은 error라도 큰 box의 경우 상대적으로 IOU에 영향을 적게 준다)

- Object가 존재하는 그리드 셀 i의 bounding box predictor j에 대해, confidence score의 loss를 계산한다 (Ci = 1)

- Object가 존재하지 않는 그리드 셀 i의 bounding box predictor j에 대해, confidence score의 loss를 계산한다 (Ci = 0)

- Object가 존재하는 그리드 셀 i에 대해, conditional class probability의 loss를 계산한다. (pi(c)=1 if class c is correct, otherwise: pi(c)=0)

- λ_coord: coordinates(x, y, w, h)에 대한 loss와 다른 loss들과의 균형을 위한 balancing parameter

- λ_noobj: 객체가 있는 box와 없는 box 간에 균형을 위한 balancing parameter

Learning techniques

1. EarlyStopping

- 성능이 더 이상 높아지지 않으면 학습을 멈춘다

2. LearningRateScheduler

- Epoch을 기준으로 learning rate를 조절한다

3. ReduceROnPlateau

- 모델의 성능이 시간이 지남에 따라 향상되지 않으면 학습률을 낮춘다

2-Stage-detector vs 1-Stage-detector

1-stage detector는 비교적 빠르지만 정확도가 낮고
2-stage detector는 비교적 느리지만 정확도가 높다

저작자표시 비영리

'Computer_Science > Visual Intelligence' 카테고리의 다른 글

33일차 - 논문 수업 (VAE) (0)	2021.11.01
32일차 - 논문 수업 (AE) (0)	2021.11.01
30일차 논문 수업 (Detection) (0)	2021.11.01
30일차 - 논문 수업 (Detection) (0)	2021.10.24
29일차 - 논문 수업 (Detection) (0)	2021.10.24

30일차 논문 수업 (Detection)

2021. 11. 1. 14:56

728x90

Object Detection

Single object detection 문제는 학습 데이터 구성을 하나의 물체로만 구성된 이미지들로 학습을 하게 되면 어렵지 않게 구현할 수 있다

하지만, Multiple object detection 문제는 물체가 한 개일 때 두 개 일 때 등 몇 개인지 정해지지 않았을 때는 문제가 까다로워진다

결국 이러한 문제를 해결하기 위해 고안된 방식이 Region proposal인 것이다

Localization이후에 detection을 하게 되면 multiple object detection 문제도 어느 정도 해결이 가능해 진다

Faster R-CNN

Fast R-CNN은 하나의 네트워크로 구성할 수 없기 때문에 병목 현상이 발생하는 문제가 있었다
이러한 문제점을 보완하여 Selective search대신 Region proposal을 네트워크로 만들어 속도를 향상시킨 모델이다

Region proposal 구하는 과정

1. Pre-trained VGG모델에 이미지를 입력값으로 넣는다

2. VGG모델로 부터 나온 feature map을 3x3 conv연산을 한다

3. class score를 매기기 위해서 feature map에 대하여 1x1 conv 연산을 한다

4. class score에 따라 상위 N개의 region proposals만을 추출한다

import tensorflow as tf 
import numpy as np 
import pandas as pd
import cv2
import matplotlib.pyplot as plt
import matplotlib.patches as pt

img = tf.keras.utils.get_file('zebra.jpg', 'https://i.imgur.com/XjeiRMV.jpg')

h = w = 800

im = cv2.imread(img)
im = cv2.cvtColor(im, cv2.COLOR_BGR2RGB)

im_r = cv2.resize(im, (h,w))
im_r_ = im_r.copy()

vgg = tf.keras.applications.VGG16(include_top=False)
backbone = tf.keras.models.Model(vgg.input, vgg.layers[17].output)

backbone(im_r_[tf.newaxis]).shape
# TensorShape([1, 50, 50, 512])

backbone(im_r_[tf.newaxis])[0,...,0].shape
# TensorShape([50, 50])

backbone(im_r_[tf.newaxis])[0,...,0]


<tf.Tensor: shape=(50, 50), dtype=float32, numpy=
array([[ 0.       ,  0.       ,  0.       , ...,  0.       ,  0.       ,
         0.       ],
       [ 0.       ,  0.       ,  0.       , ...,  0.       ,  0.       ,
         0.       ],
       [ 0.       ,  0.       ,  0.       , ...,  0.       ,  0.       ,
         0.       ],
       ...,
       [ 1.2889676, 15.054933 , 11.117959 , ...,  0.       ,  0.       ,
         0.       ],
       [ 0.       ,  0.       ,  0.       , ...,  0.       ,  0.       ,
         0.       ],
       [ 0.       ,  0.       ,  0.       , ...,  0.       ,  0.       ,
         0.       ]], dtype=float32)>

plt.imshow(backbone(im_r_[tf.newaxis])[0,...,0]) 
# 원본 이미지가 vgg네트워크를 통과하면 원본이미지의 16x16크기가 feature map의 1x1로 표현된다

plt.imshow(backbone(im_r_[tf.newaxis])[0,...,1])

plt.imshow(backbone(im_r_[tf.newaxis])[0,...,2])

IOU 계산

box1 = [20,50,200,200]
box2 = [220,220,440,440]

x = np.zeros((500,500))

def iou(box1, box2):
  x1 = max(box1[0], box2[0])
  y1 = max(box1[1], box2[1])

  x2 = min(box1[2], box2[2])
  y2 = min(box1[3], box2[3])

  if (x1 < x2 and y1 < y2):
    w_o = x2-x1 
    h_o = y2-y1 
    area =  w_o*h_o
  else:
    return 0
  
  area_b1 = (box1[2] - box1[0])*(box1[3] - box1[1])
  area_b2 = (box2[2] - box2[0])*(box2[3] - box2[1])
  union =  area_b1 + area_b2 - area

  return area/union

iou(box1, box2)
# 0

fig, ax = plt.subplots()
ax.plot()
ax.add_patch(
   pt.Rectangle(
      (box1[0], box1[1]),                   
      box1[2], box1[3],                     
      edgecolor = 'red',
      facecolor = 'yellow',
      fill=True,
   ))
ax.add_patch(
   pt.Rectangle(
      (box2[0], box2[1]),                   
      box2[2], box2[3],                     
      edgecolor = 'red',
      facecolor = 'blue',
      fill=True,
   ))
plt.show()

x = np.arange(8,800,16)
y = np.arange(8,800,16)
ratio = [0.5,1,2]
scale = [8,16,32]
al = np.zeros((22500,4)) 
count = 0
cl = np.array(np.meshgrid(x,y)).T.reshape(-1,2) 

for i in cl:
  cx, cy = i[0], i[1]
  for r in ratio:
    for s in scale:
      h = pow(pow(s,2)/r,0.5) 
      w = h*r
      h *= 16 
      w *= 16 
      xmin = cx-0.5*w
      ymin = cy-0.5*h
      xmax = cx+0.5*w
      ymax = cy+0.5*h
      al[count] = [xmin,ymin,xmax,ymax]
      count += 1

Anchor는 0보다 작고 800보다 큰 값은 제외 시키고 구한다

is_al = al[np.where((al[:,0]>=0) & (al[:,1]>=0) & (al[:,2]<=800) & (al[:,3]<=800))]
len(is_al)
# 8940

im.shape, im_r_.shape
# ((333, 500, 3), (800, 800, 3))

box1 = [120,25,200,165]
box2 = [300,50,480,320]

point = 550
h = w = 800

x = np.array([120,25,200,165])
y = np.array([300,50,480,320])


img_ = np.copy(im_r_)
for i in range(point,point+9):
  x_min = int(al[i][0])
  y_min = int(al[i][1])
  x_max = int(al[i][2])
  y_max = int(al[i][3])
  cv2.rectangle(img_, (x_min,y_min),(x_max,y_max),(0,255,0),thickness=4)

for i in range(2500):
  cv2.circle(img_, (cl[i,0],cl[i,1]), 1, (255,0,0), thickness=2)

x[0] = int(x[0]*(w/im.shape[1])) 
x[1] = int(x[1]*(h/im.shape[0]))
x[2] = int(x[2]*(w/im.shape[1]))
x[3] = int(x[3]*(h/im.shape[0]))

y[0] = int(y[0]*(w/im.shape[1]))
y[1] = int(y[1]*(h/im.shape[0]))
y[2] = int(y[2]*(w/im.shape[1]))
y[3] = int(y[3]*(h/im.shape[0]))

rec1 = cv2.rectangle(img_, (x[0],x[1]),(x[2],x[3]), color=(255,0,0), thickness=3)
rec2 = cv2.rectangle(img_, (y[0],y[1]),(y[2],y[3]), color=(255,0,0), thickness=3)

plt.imshow(img_)

x # object 1
# array([192,  60, 320, 396])

y # object2
# array([480, 120, 768, 768])

objects = [x,y]
result = np.zeros((8940,len(objects)))

for t, g in enumerate(objects):
  for i,j in enumerate(is_al):
    result[i][t] = iou(j,g)

anchor_id = np.where((al[:,0]>=0) & (al[:,1]>=0) & (al[:,2]<=800) & (al[:,3]<=800))

data = pd.DataFrame(data=[anchor_id[0],result[:,0],result[:,1]]).T # anchor당 iou값 
data.rename(columns={0:'anchor_id',1:'o1_iou',2:'o2_iou'}, inplace=True)
data.anchor_id = data.anchor_id.astype('int')

data
	anchor_id	o1_iou	o2_iou
0	1404	0.0	0.000000
1	1413	0.0	0.000000
2	1422	0.0	0.000000
3	1431	0.0	0.000000
4	1440	0.0	0.000000
...	...	...	...
8935	21051	0.0	0.065818
8936	21060	0.0	0.065818
8937	21069	0.0	0.065818
8938	21078	0.0	0.064846
8939	21087	0.0	0.058693

data.o1_iou.max(), data.o1_iou.argmax()
# (0.6562500000000001, 1785)

data.o2_iou.max(), data.o2_iou.argmax()
# (0.7119140625000001, 7540)

data[data.o1_iou > 0.65]
	anchor_id	o1_iou	o2_iou
1785	6418	0.65625	0.0
1791	6427	0.65625	0.0
2013	6868	0.65625	0.0
2019	6877	0.65625	0.0
2241	7318	0.65625	0.0
2247	7327	0.65625	0.0
2487	7768	0.65625	0.0
2493	7777	0.65625	0.0

data[data.o2_iou > 0.7]

anchor_id	o1_iou	o2_iou
7540	16877	0.0	0.711914
7547	16886	0.0	0.711914
7768	17327	0.0	0.711914
7775	17336	0.0	0.711914

data[data.o2_iou > 0.7].anchor_id.values, 1785 # 5개만 물체라고 볼수 있다 
# (array([16877, 16886, 17327, 17336]), 1785)

top = al[data[data.o2_iou > 0.7].anchor_id.values]
al[1785]
# array([ -8., 712., 120., 840.])

point = 550
h = w = 800

x = np.array([120,25,200,165])
y = np.array([300,50,480,320])


img_2 = np.copy(im_r_)
for i,j in enumerate(top):
  x_min = int(top[i][0])
  y_min = int(top[i][1])
  x_max = int(top[i][2])
  y_max = int(top[i][3])
  cv2.rectangle(img_2, (x_min,y_min),(x_max,y_max),(0,255,0),thickness=1)

# for i in range(2500):
#   cv2.circle(img_2, (cl[i,0],cl[i,1]), 1, (255,0,0), thickness=2)

x[0] = int(x[0]*(w/im.shape[1])) 
x[1] = int(x[1]*(h/im.shape[0]))
x[2] = int(x[2]*(w/im.shape[1]))
x[3] = int(x[3]*(h/im.shape[0]))

y[0] = int(y[0]*(w/im.shape[1]))
y[1] = int(y[1]*(h/im.shape[0]))
y[2] = int(y[2]*(w/im.shape[1]))
y[3] = int(y[3]*(h/im.shape[0]))

rec1 = cv2.rectangle(img_2, (x[0],x[1]),(x[2],x[3]), color=(0,0,255), thickness=3)
rec2 = cv2.rectangle(img_2, (y[0],y[1]),(y[2],y[3]), color=(0,0,255), thickness=3)

plt.figure(figsize=(10,10))
plt.imshow(img_2)

data['o1_iou_objectness'] = data.apply(lambda x: 1 if x['o1_iou'] > 0.7 else -1, axis=1)

data['o2_iou_objectness'] = data.apply(lambda x: 1 if x['o2_iou'] > 0.7 else -1, axis=1)

data[data['o1_iou_objectness']==1]

anchor_id	o1_iou	o2_iou	o1_iou_objectness

data[data['o2_iou_objectness']==1]
	anchor_id	o1_iou	o2_iou	o1_iou_objectness	o2_iou_objectness
7540	16877	0.0	0.711914	-1	1
7547	16886	0.0	0.711914	-1	1
7768	17327	0.0	0.711914	-1	1
7775	17336	0.0	0.711914	-1	1

Region proposal

input_ = tf.keras.Input((50,50,512))
x = tf.keras.layers.Conv2D(512,3, padding='same')(input_)
regressor = tf.keras.layers.Conv2D(4*9,1, activation='linear', name='reg')(x)
classifier = tf.keras.layers.Conv2D(9,1, activation='sigmoid', name='cla')(x)
PRN = tf.keras.models.Model(input_, [regressor, classifier])

PRN.summary()
Model: "model"
__________________________________________________________________________________________________
Layer (type)                    Output Shape         Param #     Connected to                     
==================================================================================================
input_2 (InputLayer)            [(None, 50, 50, 512) 0                                            
__________________________________________________________________________________________________
conv2d_2 (Conv2D)               (None, 50, 50, 512)  2359808     input_2[0][0]                    
__________________________________________________________________________________________________
reg (Conv2D)                    (None, 50, 50, 36)   18468       conv2d_2[0][0]                   
__________________________________________________________________________________________________
cla (Conv2D)                    (None, 50, 50, 9)    4617        conv2d_2[0][0]                   
==================================================================================================
Total params: 2,382,893
Trainable params: 2,382,893
Non-trainable params: 0
__________________________________________________________________________________________________

Loss

y_true = np.array([[0.,1.],[0.,0.]])
y_pred = np.array([[1.,1.],[1.,0.]])

loss1 = tf.keras.losses.MeanSquaredError()
loss2 = tf.keras.losses.mean_squared_error

tf.nn.l2_loss(y_true-y_pred)
# <tf.Tensor: shape=(), dtype=float64, numpy=1.0>

loss1(y_true, y_pred)
# <tf.Tensor: shape=(), dtype=float64, numpy=0.5>

loss2(y_true, y_pred)
# <tf.Tensor: shape=(2,), dtype=float64, numpy=array([0.5, 0.5])>

loss(y_true, y_pred)
# <tf.Tensor: shape=(), dtype=float32, numpy=0.5>

def smooth_l1_loss(y_truee, y_pred,smooth=1):
  x = tf.abs(y_true, y_pred)
  mask = tf.cast(tf.less(x,1.0), tf.float32)
  loss = (mask*(0.5*x**2)) + (1-mask)*(x-0.5)
  return loss

class MyLoss(tf.keras.losses.Loss):
  def call(self,y_true,y_pred,smooth=1):
    x = tf.abs(y_true, y_pred)
    mask = tf.cast(tf.less(x,1.0), tf.float32)
    loss = (mask*(0.5*x**2)) + (1-mask)*(smooth-0.5)
    return loss

from functools import partial
mys = partial(smooth_l1_loss, smooth=2)

저작자표시 비영리

'Computer_Science > Visual Intelligence' 카테고리의 다른 글

32일차 - 논문 수업 (AE) (0)	2021.11.01
31일차 - 논문 수업 (Detection) (0)	2021.11.01
30일차 - 논문 수업 (Detection) (0)	2021.10.24
29일차 - 논문 수업 (Detection) (0)	2021.10.24
28일차 - 논문 수업 (Detection) (0)	2021.10.24

PREV 1 2 3 4 NEXT