更新时间:2023-12-01 22:41:34
在解决问题之前,首先总结您正在使用的数据集.根据您的描述,我创建了一个示例DataFrame
可能类似于您的
Before solving the problem, let's first summarize the dataset that you're working with. Based on your description, I created an example DataFrame
that might resemble yours
import pandas as pd
dataset_size = 500
train_idx,val_idx = train_test_split(range(dataset_size),test_size=0.2,)
# create an example DataFrame that I assume will be resemble yours
example_df = pd.DataFrame({'vids':np.random.randint(0,10000,dataset_size)})
# create feature columns
for ind in range(14): example_df['feature_%i' % ind] = np.random.rand(dataset_size)
# each cell contains a list
example_df['text'] = np.random.randint(dataset_size)
example_df['text'] = example_df['text'].astype('object')
for ind in range(dataset_size):example_df.at[ind,'text'] = np.random.rand(768).tolist()
# create the label column
example_df['label'] = np.random.randint(low=0,high=5,size=dataset_size)
# extract information from the dataframe, and create data generators
all_vids = example_df['vids'].values
feature_columns = ['feature_%i' % ind for ind in range(14)]
all_features = example_df[feature_columns].values
all_text = example_df['text'].values
all_labels = example_df['label'].values
如您所见,列text
是一列列表,其中每个列表包含768个项目. labels
列包含示例的标签,无论使用单热编码还是其他类型的编码都没有关系,只要它的形状与整个神经网络模型的输出层的形状相匹配即可. vids
列是seed
列,用于动态生成随机图像.
As you can see, the column text
is a column of lists, in which each list contains 768 items. The column labels
contains the labels of the examples, it doesn't matter whether you use one-hot encoding or other types of encoding, as long as its shape matches the shape of the output layer of the overall neural network model. The column vids
is a column of seed
s for generating random images on the fly.
解决问题(基于上述数据集)
您可以对方法__getitem__
使用此语法return {'feature':features,'text':text,'vid':vid},y
,而不是堆叠三个输入数组.
You can use this syntax return {'feature':features,'text':text,'vid':vid},y
for the method __getitem__
, instead of stacking the three input arrays.
为解释这一点,让我们首先构建一个与您的玩具模型相似的玩具模型
To explain this, let's first construct a toy model resembles yours
from tensorflow.keras.models import Model
from tensorflow.keras.layers import Input,Dense,Flatten,Add
def features_part(x):
y = Dense(14)(x)
y = Dense(10,activation='linear')(y)
return y
def text_part(x):
y = Dense(768)(x)
y = Dense(10,activation='linear')(y)
return y
def vid_part(x):
y = Flatten()(x)
y = Dense(10,activation='linear')(y)
return y
input_features = Input(shape=(14,),name='feature')
input_text = Input(shape=(768,),name='text')
input_vid = Input(shape=(3,244,244,),name='vid')
feature_block = features_part(input_features)
text_block = text_part(input_text)
vid_block = vid_part(input_vid)
added = Add()([feature_block,text_block,vid_block])
# you have five classes at the end of the day
pred = Dense(1)(added)
# build model
model = Model(inputs=[input_features,input_text,input_vid],outputs=pred)
model.compile(loss='mae',optimizer='adam',metrics=['mae'])
关于此模型的最重要的事情是,我指定了三个输入层的名称
input_features = Input(shape=(14,),name='feature')
input_text = Input(shape=(768,),name='text')
input_vid = Input(shape=(3,244,244,),name='vid')
对于此模型,您可以构建类似的生成器
For this model, you can construct a generator like
# provide a seed for generating a random image
def fn2img(seed):
np.random.seed(seed)
# fake an image with three channels
return np.random.randint(low=0,high=255,size=(3,244,244))
class MultiInputDataGenerator(keras.utils.Sequence):
def __init__(self,
all_inds,labels,
features,text,vid,
shuffle=True):
self.batch_size = 8
self.labels = labels
self.all_inds = all_inds
self.shuffle = shuffle
self.on_epoch_end()
self.features = features
self.text = text
self.vid = vid
def __len__(self):
return int(np.floor(len(self.all_inds) / self.batch_size))
def __getitem__(self,index):
indexes = self.indexes[index*self.batch_size:(index+1)*self.batch_size]
batch_indices = [self.all_inds[k] for k in indexes]
features,text,vid,y = self.__data_generation(batch_indices)
return {'feature':features,'text':text,'vid':vid},y
def on_epoch_end(self):
self.indexes = np.arange(len(self.all_inds))
if self.shuffle == True:
np.random.shuffle(self.indexes)
def __data_generation(self,batch_indices):
# Generate data
features = self.features[batch_indices,:]
# note that you need to stack the slice in order to reshape it to (num_samples,768)
text = np.stack(self.text[batch_indices])
# since batch_size is not a super large number, you can stack here
vid = np.stack([fn2img(seed) for seed in self.vid[batch_indices]])
y = self.labels[batch_indices]
return features,text,vid,y
如您所见,
__getitem__
方法返回字典{'feature':features,'text':text,'vid':vid},y
.字典的键与三个输入层的名称匹配.而且,随机图像是动态生成的.
as you can see, the __getitem__
method returns a dictionary {'feature':features,'text':text,'vid':vid},y
. The keys of the dictionary match the names of the three input layers. Moreover, the random images are generated on the fly.
为了确保一切正常,您可以运行以下脚本,
In order to make sure everything works, you can run the script below,
import numpy as np
import pandas as pd
from tensorflow import keras
from sklearn.model_selection import train_test_split
from tensorflow.keras.models import Model
from tensorflow.keras.layers import Input,Dense,Flatten,Add
# provide a seed for generating a random image
def fn2img(seed):
np.random.seed(seed)
# fake an image with three channels
return np.random.randint(low=0,high=255,size=(3,244,244))
class MultiInputDataGenerator(keras.utils.Sequence):
def __init__(self,
all_inds,labels,
features,text,vid,
shuffle=True):
self.batch_size = 8
self.labels = labels
self.all_inds = all_inds
self.shuffle = shuffle
self.on_epoch_end()
self.features = features
self.text = text
self.vid = vid
def __len__(self):
return int(np.floor(len(self.all_inds) / self.batch_size))
def __getitem__(self,index):
indexes = self.indexes[index*self.batch_size:(index+1)*self.batch_size]
batch_indices = [self.all_inds[k] for k in indexes]
features,text,vid,y = self.__data_generation(batch_indices)
return {'feature':features,'text':text,'vid':vid},y
def on_epoch_end(self):
self.indexes = np.arange(len(self.all_inds))
if self.shuffle == True:
np.random.shuffle(self.indexes)
def __data_generation(self,batch_indices):
# Generate data
features = self.features[batch_indices,:]
# note that you need to stack the slice in order to reshape it to (num_samples,768)
text = np.stack(self.text[batch_indices])
# since batch_size is not a super large number, you can stack here
vid = np.stack([fn2img(seed) for seed in self.vid[batch_indices]])
y = self.labels[batch_indices]
return features,text,vid,y
# fake a dataset
dataset_size = 500
train_idx,val_idx = train_test_split(range(dataset_size),test_size=0.2,)
# create an example DataFrame that I assume will be resemble yours
example_df = pd.DataFrame({'vids':np.random.randint(0,10000,dataset_size)})
# create feature columns
for ind in range(14): example_df['feature_%i' % ind] = np.random.rand(dataset_size)
# each cell contains a list
example_df['text'] = np.random.randint(dataset_size)
example_df['text'] = example_df['text'].astype('object')
for ind in range(dataset_size):example_df.at[ind,'text'] = np.random.rand(768).tolist()
# create the label column
example_df['label'] = np.random.randint(low=0,high=5,size=dataset_size)
# extract information from the dataframe, and create data generators
all_vids = example_df['vids'].values
feature_columns = ['feature_%i' % ind for ind in range(14)]
all_features = example_df[feature_columns].values
all_text = example_df['text'].values
all_labels = example_df['label'].values
training_generator = MultiInputDataGenerator(train_idx,all_labels,all_features,all_text,all_vids)
# create model
def features_part(x):
y = Dense(14)(x)
y = Dense(10,activation='linear')(y)
return y
def text_part(x):
y = Dense(768)(x)
y = Dense(10,activation='linear')(y)
return y
def vid_part(x):
y = Flatten()(x)
y = Dense(10,activation='linear')(y)
return y
input_features = Input(shape=(14,),name='feature')
input_text = Input(shape=(768,),name='text')
input_vid = Input(shape=(3,244,244,),name='vid')
feature_block = features_part(input_features)
text_block = text_part(input_text)
vid_block = vid_part(input_vid)
added = Add()([feature_block,text_block,vid_block])
# you have five classes at the end of the day
pred = Dense(1)(added)
# build model
model = Model(inputs=[input_features,input_text,input_vid],outputs=pred)
model.compile(loss='mae',optimizer='adam',metrics=['mae'])
model.fit_generator(generator=training_generator,epochs=10)
print(model.history.history)